Updating a large number of entities in a datastore on Google App Engine

2024/10/7 10:21:11

I would like to perform a small operation on all entities of a specific kind and rewrite them to the datastore. I currently have 20,000 entities of this kind but would like a solution that would scale to any amount.

What are my options?

Answer

Use a mapper - this is part of the MapReduce framework, but you only want the first component, map, as you don't need the shuffle/reduce step if you're simply mutating datastore entities.

https://en.xdnf.cn/q/70253.html

Related Q&A

Is there a neater alternative to `except: pass`?

I had a function that returned a random member of several groups in order of preference. It went something like this:def get_random_foo_or_bar():"Id rather have a foo than a bar."if there_are…

Get a permutation as a function of a unique given index in O(n)

I would like to have a function get_permutation that, given a list l and an index i, returns a permutation of l such that the permutations are unique for all i bigger than 0 and lower than n! (where n …

How to generate Bitcoin keys/addresses from a seed in Python?

I am trying to create a set of public/private keys from a mnemonic based on BIP0039. I am working in Python.Here is the code I have so far:from mnemonic import Mnemonic mnemon = Mnemonic(english) words…

NumPy - Set values in structured array based on other values in structured array

I have a structured NumPy array:a = numpy.zeros((10, 10), dtype=[("x", int),("y", str)])I want to set values in a["y"] to either "hello" if the corresponding val…

numpy.disutils.system_info.NotFoundError: no lapack/blas resources found

Problem: Linking numpy to correct Linear Algebra libraries. Process is so complicated that I might be looking for the solution 6th time and I have no idea whats going wrong. I am on Ubuntu 12.04.5. I …

Opencv: Jetmap or colormap to grayscale, reverse applyColorMap()

To convert to colormap, I doimport cv2 im = cv2.imread(test.jpg, cv2.IMREAD_GRAYSCALE) im_color = cv2.applyColorMap(im, cv2.COLORMAP_JET) cv2.imwrite(colormap.jpg, im_color)Then,cv2.imread(colormap.jpg…

Get file modification time to nanosecond precision

I need to get the full nanosecond-precision modified timestamp for each file in a Python 2 program that walks the filesystem tree. I want to do this in Python itself, because spawning a new subprocess …

`TypeError: argument 2 must be a connection, cursor or None` in Psycopg2

I have a heroku pipeline set up, and have just enabled review apps for it. It is using the same codebase as my staging and production apps, same settings files and everything.When the review app spins …

replace string if length is less than x

I have a dataframe below. a = {Id: [ants, bees, cows, snakes, horses], 2nd Attempts: [10, 12, 15, 14, 0],3rd Attempts: [10, 10, 9, 11, 10]} a = pd.DataFrame(a) print (a)I want to able add text (-s) to …

Convert ast node into python object

Given an ast node that can be evaluated by itself, but is not literal enough for ast.literal_eval e.g. a list comprehensionsrc = [i**2 for i in range(10)] a = ast.parse(src)Now a.body[0] is an ast.Expr…