Python seems to use a lot of memory. So what exactly is the overhead of each type of value? Short answer:
int | 24 |
float | 24 |
tuple | 63 |
list | 101 |
dict | 298 |
old-style class | 345 |
new-style class | 336 |
subclassed tuple | 79 |
Record | 79 |
Record with old class mixin | 79 |
Record with new class mixin | 79 |
I measured these by running a simple program that loaded up 1,000,000 values in a list and then did time.sleep(1000). I ran that for different value types and then ran "top" to see how much memory was being used. I took that value, substracted the memory usage for a list of all the same value (14 bytes each), subtracted the value of a child value (usually an int, 24 bytes each), and then divided by 1,000,000. I'll include the code I ran at the end if you want to cut and paste. So what lessons do we learn from this?
- Python objects are very expensive at over 300 bytes each.
- Tuples have 1/5 as much overhead.
- Records are almost as good as tuples, even when a mixin is added.
So, if you want to have lots of values in memory without using lots of memory, use Record.
If you want to run the test for yourself, here's the code. Just comment out the "make_val" that you want to test.
import time from Record import Record class TupleClass(tuple): pass class RecordClass(Record("val")): pass class OldClass: def __init__(self, val): self.val = val def method(self): pass class NewClass(object): def __init__(self, val): self.val = val def method(self): pass class RecordWithOldClass(Record("val"), OldClass): pass class RecordWithNewClass(Record("val"), NewClass): pass make_val = lambda i : 1 #nothing (base overhead) #make_val = lambda i : i #make_val = float #make_val = lambda i : (i,) #make_val = lambda i : [i] #make_val = lambda i : {i:i} #make_val = TupleClass #make_val = RecordClass #make_val = OldClass #make_val = NewClass #make_val = RecordWithOldClass #make_val = RecordWithNewClass count = 1000000 lst = [make_val(i) for i in xrange(count)] time.sleep(100000)
Fun - I was just playing with the same sort of thing and came to a similar conclusion. One item that's useful is a slotted new style object:
ReplyDeleteclass Foo(object):
__slots__ = []
These come out to be (on average) slightly greater than 16 bytes / each.
If you add values:
class Foo(object):
__slots__ = ['a','b']
def __init__(self):
self.a = None
self.b = None
They come out to be ~65 bytes each.
Wow Gary! I tried that too and it came to about a quarter the size of virtual memory. This is way over my head but clearly indicates that __slots__ might be the way forward when memory matters.
ReplyDeleteIt would be interesting to see the exact same test run with a new-style class with __slots__ to eliminate the __dict__ attribute. According to your chart, the instance takes up 345 bytes, but the __dict__ should be taking up 298 of that; that that is eliminated when you use __slots__.
ReplyDeleteOops, I should have read the preceding comments before posting mine! Sorry.
ReplyDeleteI am getting wildly bigger values. Try this:
ReplyDelete#!/bin/sh
(
exec >test.py
echo "\
#!/usr/bin/python
import time"
i=300000
while test $i != 0; do
echo "i$i=$i"
: $((i--))
done
echo "time.sleep(3)"
)
chmod 755 test.py
echo "Before: `grep ^MemFree: /proc/meminfo`"
./test.py &
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
sleep 1
echo "During: `grep ^MemFree: /proc/meminfo`"
wait
echo "After: `grep ^MemFree: /proc/meminfo`"
# i=100000, output on my machine (x86-64):
#Before: MemFree: 308688 kB
#During: MemFree: 252144 kB
#During: MemFree: 227460 kB
#During: MemFree: 227468 kB
#During: MemFree: 227460 kB
#After: MemFree: 308200 kB
# Thus, (308200-227460)/100 = 807 bytes per each int variable
# i=300000
#Before: MemFree: 1007572 kB
#During: MemFree: 695952 kB
#During: MemFree: 421548 kB
#During: MemFree: 851084 kB
#During: MemFree: 835708 kB
#During: MemFree: 808800 kB
#During: MemFree: 795656 kB
#During: MemFree: 795656 kB
#During: MemFree: 795648 kB
#During: MemFree: 795656 kB
#After: MemFree: 1007448 kB
# Thus, (1007448-795648)/300 = 706 bytes per each int variable
I am a big fan of your blog.i am so excited by a read of your blog's content. great post.Thanks for sharing superb information.
ReplyDeleteProWeb365 Minneapolis web design
Mua vé máy bay tại Aivivu, tham khảo
ReplyDeletevé máy bay đi Mỹ tháng nào rẻ nhất
giá vé máy bay từ mỹ về việt nam
vé máy bay từ canada về việt nam
vé máy bay từ hàn quốc về việt nam
I really appreciate the information that you have shared on your Blog. Thanks for shearing this blog.
ReplyDeleteprinter repair cincinnati
Uninstall HP Print And Scan Doctor
Printer Repair Fort Collins
How Do I Get My HP Envy 4500 Printer Back Online
how to fix damaged canon ink cartridge
How to Resolve HP Printer Error 0xc05d1281
Why Wireless HP Printer Not Connecting Automatically Windows 10
Do you know the net worth of your favorite actor, singer or any celebrity? There is a website which contains all information about celeb networth for everyone.
ReplyDeleteThanks for sharing this informative content.,
ReplyDeleteLeanpitch provides online training in Scrum Master Certification during this lockdown period everyone can use it wisely.
Join Leanpitch 2 Days CSM Certification Workshop in different cities.
CSM online
CSM online certification
ReplyDeleteThanks for sharing this informative content.,
Leanpitch provides online training in Scrum Master Certification during this lockdown period everyone can use it wisely.
Join Leanpitch 2 Days CSM Certification Workshop in different cities.
CSM online training
CSM training online
ReplyDeleteThanks for sharing this informative content.,
Leanpitch provides online training in Scrum Master Certification during this lockdown period everyone can use it wisely.
Join Leanpitch 2 Days CSM Certification Workshop in different cities.
CSM training online
Scrum master training online
Hey, Your blog is very informative. It is nice to read such high-quality content. Attractive information on your blog, thank you for taking the time and share with us. The experts of our Who can write my essay for me provide solution regarding paperhelp within the time that you provide us. Myassignmenthelp delivers high-quality content related to proposal essay topics and research paper help.
ReplyDeleteI love it when people get together and share views. Great site, Eunidrip company has wealth of experience in drilling boreholes.
ReplyDeleteThis post is quite informative. I love reading your blogs. Quite helpful. Are you also searching for Academic content writing services we are the best solution for you. We are best known for delivering the best services to students without having to break the bank.
ReplyDeleteI love this. It is soo informative. Are you also searching for Swedish assignment writing help we are the best solution for you. We are best known for delivering the best services to students without having to break the bank
ReplyDeleteAwesome article like your previous article Cheap Finance Assignment! I want people to know just how good this information is in your article. It’s interesting, compelling content. Your views are much like my own concerning this subject.
ReplyDeleteThank you for sharing this blog . your blog is very interesting and helpful. Quickbooks is a accounting software which is used in many organization to manage daily data. Quickbooks also available with many versions. Sometimes when Quickbooks's file get damage or corrupt it stops working. This error is called Quickbooks error 1334 . Due to this error all work get stuck.
ReplyDeleteran that for different value types and then ran top to see. Buy A Custom Essay took that value, substracted the memory usage for a list of all the same value.
ReplyDeleteThanks for sharing such a great blog Keep posting.
ReplyDeleteGreat article
ReplyDeletepaybyplate ma pay online
watch all episode of pinoy media pinoy tv and pinoy tambayan on our website
ReplyDeletehttps://krogerexperiencee
Nice Blog. QuickBooks Error 15241 is an error that prevents the QuickBooks Desktop software from successfully installing updates. It is connected to the QuickBooks Payroll feature. It's most likely caused by the QuickBooks Desktop software's File Copy Service being turned off.
ReplyDeleteI find all of this information fascinating, and it is full of useful information, so I'll be following your blog to learn new things.
ReplyDeleteHey thanks for this informative post, if you by any chance face QuickBooks Pro Error 1334 in your Quickbooks accounting software, any types of network issues or company file issues make sure to visit ebetterbooks.
ReplyDeleteWynn casino opens in Las Vegas - FilmfileEurope
ReplyDeleteWynn's first hotel casino in Las Vegas since opening its doors in 1996, 출장안마 Wynn Las Vegas febcasino is the first hotel on the Strip to https://septcasino.com/review/merit-casino/ offer such a large 토토 사이트 selection worrione of
virtual accounting firm provide their clients with real-time access to business accounting records by utilising Cloud Accounting software such as Xero. Indeed, platforms such as Xero enable virtual accountants and their clients to collaborate on their accounting records even if they are not in the same city, town, or state.
ReplyDeletevirtual accounting firm provide their clients with real-time access to business accounting records by utilising Cloud Accounting software such as Xero. Indeed, platforms such as Xero enable virtual accountants and their clients to collaborate on their accounting records even if they are not in the same city, town, or state.
ReplyDelete