NumPy arrays with SQLite

2024/12/9 5:26:09

The most common SQLite interface I've seen in Python is sqlite3, but is there anything that works well with NumPy arrays or recarrays? By that I mean one that recognizes data types and does not require inserting row by row, and extracts into a NumPy (rec)array...? Kind of like R's SQL functions in the RDB or sqldf libraries, if anyone is familiar with those (they import/export/append whole tables or subsets of tables to or from R data tables).

Answer

why not give redis a try?

Drivers for your two platforms of interest are available--python (redis, via package index]2), and R (rredis, CRAN).

The genius of redis is not that it will magically recognize the NumPy data type and allow you to insert and extract multi-dimensional NumPy arrays as if they were native redis datatypes, rather its genius is in the remarkable ease with which you can create such an interface with just a few lines of code.

There are (at least) several tutorials on redis in python; the one on the DeGizmo blog is particularly good.

import numpy as NP# create some data
A = NP.random.randint(0, 10, 40).reshape(8, 5)# a couple of utility functions to (i) manipulate NumPy arrays prior to insertion 
# into redis db for more compact storage & 
# (ii) to restore the original NumPy data types upon retrieval from redis db
fnx2 = lambda v : map(int, list(v))
fnx = lambda v : ''.join(map(str, v))# start the redis server (e.g. from a bash prompt)
$> cd /usr/local/bin      # default install directory for 'nix
$> redis-server           # starts the redis server# start the redis client:
from redis import Redis
r0 = Redis(db=0, port=6379, host='localhost')       # same as: r0 = Redis()# to insert items using redis 'string' datatype, call 'set' on the database, r0, and
# just pass in a key, and the item to insert
r0.set('k1', A[0,:])# row-wise insertion the 2D array into redis, iterate over the array:
for c in range(A.shape[0]):r0.set( "k{0}".format(c), fnx(A[c,:]) )# or to insert all rows at once
# use 'mset' ('multi set') and pass in a key-value mapping: 
x = dict([sublist for sublist in enumerate(A.tolist())])
r0.mset(x1)# to retrieve a row, pass its key to 'get'
>>> r0.get('k0')'63295'# retrieve the entire array from redis:
kx = r0.keys('*')           # returns all keys in redis database, r0for key in kx :r0.get(key)# to retrieve it in original form:
A = []
for key in kx:A.append(fnx2(r0.get("{0}".format(key))))>>> A = NP.array(A)
>>> Aarray([[ 6.,  2.,  3.,  3.,  9.],[ 4.,  9.,  6.,  2.,  3.],[ 3.,  7.,  9.,  5.,  0.],[ 5.,  2.,  6.,  3.,  4.],[ 7.,  1.,  5.,  0.,  2.],[ 8.,  6.,  1.,  5.,  8.],[ 1.,  7.,  6.,  4.,  9.],[ 6.,  4.,  1.,  3.,  6.]])
https://en.xdnf.cn/q/73391.html

Related Q&A

Binary Phase Shift Keying in Python

Im currently working on some code to transmit messages/files/and other data over lasers using audio transformation. My current code uses the hexlify function from the binascii module in python to conve…

Django. Listing files from a static folder

One seemingly basic thing that Im having trouble with is rendering a simple list of static files (say the contents of a single repository directory on my server) as a list of links. Whether this is sec…

inconsistent migration history when changing a django apps name

Im trying to rename one of the apps in my django website. There is another app which depends on it and its mysql tables. I went over all the files in both apps and changed the instances of the old name…

Tensorflow vs Numpy math functions

Is there any real difference between the math functions performed by numpy and tensorflow. For example, exponential function, or the max function? The only difference I noticed is that tensorflow take…

PyQt5: I cant understand QGraphicsScenes setSceneRect(x, y, w, h)

I see some people say if you want to put QGraphicsScenes origin of coordinates at the origin of QGraphicsView, i.e. top-left corner. You need to let both of them have the same size.So here is what I do…

Remove first character from string Django template

I know this has been asked multiple times but the solution that everyone reaches (and the documentation) doesnt seem to be working for me...Trying to remove first characterCode is {{ picture.picture_pa…

Prevent Python logger from printing to console

Im getting mad at the logging module from Python, because I really have no idea anymore why the logger is printing out the logging messages to the console (on the DEBUG level, even though I set my File…

How to remove python assertion when compiling in cython?

so, here is my problem: I code in python, but I need to improve performance in some part of my code that are too slow. A good(and easy) solution seems to be using cython; I tried it and got good result…

ignoring newline character in regex match

I am trying to replace all matching occurrences with title cases using the following script. When there is a newline character between filter words (in this case ABC and DEF) that line doesnt get repla…

Xml parsing from web response

Im trying to get response from nominatim to geo-code few thousands of cities. import os import requests import xml.etree.ElementTree as ETtxt = open(input.txt, r).readlines() for line in txt:lp, region…