I've learned. I'll share.

October 14, 2008

Python Memory Usage: What values are taking up so much memory?

Python seems to use a lot of memory. So what exactly is the overhead of each type of value? Short answer:

int 24
float 24
tuple 63
list 101
dict 298
old-style class 345
new-style class 336
subclassed tuple 79
Record 79
Record with old class mixin 79
Record with new class mixin 79
Measured in bytes using Python 2.5 in 64-bit Ubuntu Linux

I measured these by running a simple program that loaded up 1,000,000 values in a list and then did time.sleep(1000). I ran that for different value types and then ran "top" to see how much memory was being used. I took that value, substracted the memory usage for a list of all the same value (14 bytes each), subtracted the value of a child value (usually an int, 24 bytes each), and then divided by 1,000,000. I'll include the code I ran at the end if you want to cut and paste. So what lessons do we learn from this?

  • Python objects are very expensive at over 300 bytes each.
  • Tuples have 1/5 as much overhead.
  • Records are almost as good as tuples, even when a mixin is added.

So, if you want to have lots of values in memory without using lots of memory, use Record.

If you want to run the test for yourself, here's the code. Just comment out the "make_val" that you want to test.

import time
from Record import Record

class TupleClass(tuple):
    pass

class RecordClass(Record("val")):
    pass

class OldClass:
    def __init__(self, val):
        self.val = val

    def method(self):
        pass

class NewClass(object):
    def __init__(self, val):
        self.val = val

    def method(self):
        pass

class RecordWithOldClass(Record("val"), OldClass):
    pass

class RecordWithNewClass(Record("val"), NewClass):
    pass

make_val = lambda i : 1 #nothing (base overhead)
#make_val = lambda i : i    
#make_val = float           
#make_val = lambda i : (i,) 
#make_val = lambda i : [i]  
#make_val = lambda i : {i:i}
#make_val = TupleClass
#make_val = RecordClass
#make_val = OldClass
#make_val = NewClass
#make_val = RecordWithOldClass
#make_val = RecordWithNewClass

count = 1000000
lst = [make_val(i) for i in xrange(count)]
time.sleep(100000)

30 comments:

  1. Fun - I was just playing with the same sort of thing and came to a similar conclusion. One item that's useful is a slotted new style object:

    class Foo(object):
      __slots__ = []

    These come out to be (on average) slightly greater than 16 bytes / each.

    If you add values:

    class Foo(object):
      __slots__ = ['a','b']
      def __init__(self):
        self.a = None
        self.b = None

    They come out to be ~65 bytes each.

    ReplyDelete
  2. Wow Gary! I tried that too and it came to about a quarter the size of virtual memory. This is way over my head but clearly indicates that __slots__ might be the way forward when memory matters.

    ReplyDelete
  3. It would be interesting to see the exact same test run with a new-style class with __slots__ to eliminate the __dict__ attribute. According to your chart, the instance takes up 345 bytes, but the __dict__ should be taking up 298 of that; that that is eliminated when you use __slots__.

    ReplyDelete
  4. Oops, I should have read the preceding comments before posting mine! Sorry.

    ReplyDelete
  5. I am getting wildly bigger values. Try this:

    #!/bin/sh
    (
    exec >test.py
    echo "\
    #!/usr/bin/python
    import time"
    i=300000
    while test $i != 0; do
    echo "i$i=$i"
    : $((i--))
    done
    echo "time.sleep(3)"
    )

    chmod 755 test.py
    echo "Before: `grep ^MemFree: /proc/meminfo`"
    ./test.py &
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    sleep 1
    echo "During: `grep ^MemFree: /proc/meminfo`"
    wait
    echo "After: `grep ^MemFree: /proc/meminfo`"

    # i=100000, output on my machine (x86-64):
    #Before: MemFree: 308688 kB
    #During: MemFree: 252144 kB
    #During: MemFree: 227460 kB
    #During: MemFree: 227468 kB
    #During: MemFree: 227460 kB
    #After: MemFree: 308200 kB
    # Thus, (308200-227460)/100 = 807 bytes per each int variable

    # i=300000
    #Before: MemFree: 1007572 kB
    #During: MemFree: 695952 kB
    #During: MemFree: 421548 kB
    #During: MemFree: 851084 kB
    #During: MemFree: 835708 kB
    #During: MemFree: 808800 kB
    #During: MemFree: 795656 kB
    #During: MemFree: 795656 kB
    #During: MemFree: 795648 kB
    #During: MemFree: 795656 kB
    #After: MemFree: 1007448 kB
    # Thus, (1007448-795648)/300 = 706 bytes per each int variable

    ReplyDelete
  6. I am a big fan of your blog.i am so excited by a read of your blog's content. great post.Thanks for sharing superb information.
    ProWeb365 Minneapolis web design

    ReplyDelete

  7. Thanks for sharing this informative content.,
    Leanpitch provides online training in Scrum Master Certification during this lockdown period everyone can use it wisely.
    Join Leanpitch 2 Days CSM Certification Workshop in different cities.
    CSM online training

    CSM training online

    ReplyDelete

  8. Thanks for sharing this informative content.,
    Leanpitch provides online training in Scrum Master Certification during this lockdown period everyone can use it wisely.
    Join Leanpitch 2 Days CSM Certification Workshop in different cities.
    CSM training online

    Scrum master training online

    ReplyDelete
  9. Hey, Your blog is very informative. It is nice to read such high-quality content. Attractive information on your blog, thank you for taking the time and share with us. The experts of our Who can write my essay for me provide solution regarding paperhelp within the time that you provide us. Myassignmenthelp delivers high-quality content related to proposal essay topics and research paper help.

    ReplyDelete
  10. I love it when people get together and share views. Great site, Eunidrip company has wealth of experience in drilling boreholes.

    ReplyDelete
  11. This post is quite informative. I love reading your blogs. Quite helpful. Are you also searching for Academic content writing services we are the best solution for you. We are best known for delivering the best services to students without having to break the bank.

    ReplyDelete
  12. I love this. It is soo informative. Are you also searching for Swedish assignment writing help we are the best solution for you. We are best known for delivering the best services to students without having to break the bank

    ReplyDelete
  13. Awesome article like your previous article Cheap Finance Assignment! I want people to know just how good this information is in your article. It’s interesting, compelling content. Your views are much like my own concerning this subject.

    ReplyDelete
  14. Thank you for sharing this blog . your blog is very interesting and helpful. Quickbooks is a accounting software which is used in many organization to manage daily data. Quickbooks also available with many versions. Sometimes when Quickbooks's file get damage or corrupt it stops working. This error is called Quickbooks error 1334 . Due to this error all work get stuck.

    ReplyDelete
  15. ran that for different value types and then ran top to see. Buy A Custom Essay took that value, substracted the memory usage for a list of all the same value.

    ReplyDelete
  16. Thanks for sharing such a great blog Keep posting.

    ReplyDelete
  17. watch all episode of pinoy media pinoy tv and pinoy tambayan on our website

    https://krogerexperiencee

    ReplyDelete
  18. Nice Blog. QuickBooks Error 15241 is an error that prevents the QuickBooks Desktop software from successfully installing updates. It is connected to the QuickBooks Payroll feature. It's most likely caused by the QuickBooks Desktop software's File Copy Service being turned off.

    ReplyDelete
  19. I find all of this information fascinating, and it is full of useful information, so I'll be following your blog to learn new things.

    ReplyDelete
  20. Hey thanks for this informative post, if you by any chance face QuickBooks Pro Error 1334 in your Quickbooks accounting software, any types of network issues or company file issues make sure to visit ebetterbooks.

    ReplyDelete
  21. Wynn casino opens in Las Vegas - FilmfileEurope
    Wynn's first hotel casino in Las Vegas since opening its doors in 1996, 출장안마 Wynn Las Vegas febcasino is the first hotel on the Strip to https://septcasino.com/review/merit-casino/ offer such a large 토토 사이트 selection worrione of

    ReplyDelete
  22. virtual accounting firm provide their clients with real-time access to business accounting records by utilising Cloud Accounting software such as Xero. Indeed, platforms such as Xero enable virtual accountants and their clients to collaborate on their accounting records even if they are not in the same city, town, or state.

    ReplyDelete
  23. virtual accounting firm provide their clients with real-time access to business accounting records by utilising Cloud Accounting software such as Xero. Indeed, platforms such as Xero enable virtual accountants and their clients to collaborate on their accounting records even if they are not in the same city, town, or state.

    ReplyDelete
  24. I got many information in this blog keep share more useful information thanks. and we give best legal support for many people if anyone want it kindly visit our site thanks. reckless driving lawyer Henrico va

    ReplyDelete

  25. I recently completed my first scarf, and I couldn't be more proud of it! With each stitch, I felt a sense of accomplishment and creativity.
    New Jersey Federal Criminal Lawyer
    Federal Criminal Defense Lawyer

    ReplyDelete
  26. Motorcycle accident lawyers assist clients in navigating insurance claims, gathering evidence, and pursuing compensation for medical expenses, property damage, and other losses resulting from the accident. motorcycle accident lawyers

    ReplyDelete
  27. Fascinating breakdown of Python memory usage! The insights into different value types and their overheads are invaluable. Thanks for sharing this detailed analysis and code snippet for testing
    Driving Without A License In New Jersey

    ReplyDelete
  28. Python's memory usage can often be a mystery, especially when dealing with large datasets or complex algorithms. By delving into memory profiling, one can uncover valuable insights into which variables, data structures, or operations are consuming significant memory.

    ReplyDelete

Blog Archive

Google Analytics