Wes McKinney's Blog

Re “hopefully an R guru can tell me how to deal with this trailing delimiter crap”. I’m not a guru but what’s wrong with reading in the final blank column, then removing that column once it’s in? Yes, as you say, it is tedious to specify column types in advance in R. So you can read the first few rows to get the types, than pass that back in to colClasses. To say that R “deeply struggles” is going a little too far. Your file reader is wonderful, yes. It is better, more convenient, than R’s. But do you really have to bash R so harshly?

[Reply]

Wes McKinney Reply:
December 11th, 2012 at 3:28 pm

Sorry, I softened the language from “deeply struggles” to “makes it more difficult”. Have you tried loading this file in R? The issue is that the header column does NOT have a trailing delimiter so R wants to use the first column as the row names. If you pass row.names=NULL it adds a row.names first column name and mislabels the other columns.

If someone from the R community gets motivated, the file tokenizer I built (the interesting part) is liberally licensed C code and could be easily adapted to R. I’m not much for the language wars and think that everyone should have fast, easy to use data processing tools in all languages. You’re one of the only people I know who’s been making major strides on this front recently (really great work with data.table).

[Reply]

Matthew Dowle Reply:
December 11th, 2012 at 4:55 pm

Great, thanks. I see what you mean about the header row now. I haven’t tried it, but how about skip=1,header=FALSE to read the data (only) ignoring the header row. Then read the header row separately (nrows=1) and assign the row names afterwards. A faff, though. Interesting case.

Funny you should say that. I’ve been working on a faster and friendlier file reader for R in spare time. There’s still quite a bit to do, but an early draft of its help file with some examples and timings is below. One of the things it’ll read directly is dual-delimited files, where each cell of a list column is itself a vector, such as in genomics (a second separator, within some columns). I’ll certainly take a look at your tokenizer, thanks. Also your use of mmap looks neat.

https://r-forge.r-project.org/scm/viewvc.php/pkg/man/fread.Rd?view=markup&root=datatable

[Reply]

Excellent work. It’s great to see vbench demonstrate just how far you guys are coming along.

I just tried to use 0.9.1′s read_csv on my machine for this FEC dataset, it used 8Gb of swap! (I killed it after it gave no result after an hour.)

[Reply]

I would prefer that pandas beat the R read.csv in all respects. Hopefully this can be added to improve the robustness of read_csv.

http://stackoverflow.com/questions/13824840/feature-request-for-pandas-read-csv

[Reply]

Wes McKinney Reply:
December 11th, 2012 at 5:22 pm

Just replied. You have to specify the escape character to make that case work. R has a default option that quotes are coded as `”` which makes it work out of the box in your case. Maybe that should be the default for pandas, too.

[Reply]

Hi, Hi! Sorry to bug you about this. I am new to python and pandas. I started coding in Python 3 and later discovered all the distributions like python(x,y), and the fact that they are all using 2.7. Moving to 2.7 seems odd now. I’ve been able to install all recomended packages/soft like Ipython etc on win7-64 using python 3.2, except for PyTables. By any chance are you aware if they are planning to release a version for Python 3? I searched their site for way to contact PyTables developers, but failed…

[Reply]

Wes McKinney Reply:
January 2nd, 2013 at 8:07 pm

I’d suggest you stick to 2.7 for another year or so if you need PyTables. I haven’t looked to see how much work would be involved in making the C code there 2 and 3 compatible.

[Reply]

it’s really good, thanks

[Reply]

First and foremost, pandas is the coolest thing I’ve found in a long time. Many thanks.

Is it expected that read_csv is much faster than to_csv ?

I’m seeing read_csv : to_csv : (writing to hdf5) at about 1 : 50 : 2.5

[Reply]

Wes McKinney Reply:
January 29th, 2013 at 2:47 am

Yes, to_csv is much slower and does not have any optimized code. I would be interested to see this changed but do not have the time to implement it myself for some time.

[Reply]

I tried this on my Mac with 8GB memory, but I got this:

Python(12618,0xacb01a28) malloc: *** mmap(size=24051712) failed (error code=12)
*** error: can’t allocate region
*** set a breakpoint in malloc_error_break to debug

what is possible reason for this? thanks for your help

[Reply]

Wes McKinney Reply:
February 8th, 2013 at 5:45 pm

Can you post this on GitHub please? https://github.com/pydata/pandas/issues?state=open

[Reply]

It’s great to see read_csv get such a big improvement. I’m curious as to why or the short version of why read_csv is so much faster than numpy.loadxt()?

For example, I’m comparing numbers on a relatively small space-separated file (about 208K) with 26 columns and 1000 rows. Each item in the file is a random floating point number that I generated with numpy.rand.

>>> %timeit f = open(‘test.out’, ‘r’);f.readline();np.loadtxt(f, unpack=True)
10 loops, best of 3: 26.9 ms per loop

>>> %timeit pd.read_csv(‘test.out’, delim_whitespace=True)
100 loops, best of 3: 5.99 ms per loop

[Reply]

Anonymous Reply:
March 8th, 2013 at 3:49 pm

Guess the answer is pretty much already on your blog! For others to reference:

- Fast new file parser added: http://wesmckinney.com/blog/?m=201210

- Speeding up file parsing with Cython: http://wesmckinney.com/blog/?p=278

[Reply]

Wes McKinney Reply:
March 8th, 2013 at 3:53 pm

Short answer: file tokenization and type inference is being handled at the lowest level possible in C/Cython. If you look at the impl of numpy.loadtxt you’ll see a lot of Python.

[Reply]

Anonymous Reply:
March 8th, 2013 at 4:42 pm

Oh ok, that makes a lot of sense. I did look into read_csv but not loadtxt. Thanks for the feedback!

[Reply]