ArcGIS for Desktop
Some general tips for making calculations faster:
- Be successful overlaying large, complex datasets in Geoprocessing, good advice also for other geoprocessing tools.
- Use ArcGIS Pro 1.4 OR ArcGIS Server 10.5 OR ArcMAP with ArcGIS for Desktop Background Geoprocessing (64 bit). Using 64 bit processing to perform analysis on systems with large amounts of RAM may help when processing large data which may have otherwise failed in a 32 bit environment.
- Using the in_memory workspace instead of using geometry objects is faster, but if you need memory also for calculation and have big dataset there might not be enough memory available.
- Python Multiprocessing – Approaches and Considerations
- Blog: Python, ArcGIS and multiprocessing
- Parallel Python: Multiprocessing With ArcPy, examples
- Example: Producing Building Shadows
- Multiprocessing Basics
Some tips from these sites:
- Each process locks the used data, so only one process can access data from same file/FileGDB.
- Multiprocessing is possible, if you are using several files (Shape, tiff etc) or suitable database.
- If you want to use multiprocessing for geocoding or routing, you need several copies of the road network data.
- For each process, there is a start-up cost loading the arcpy library (1-3 seconds). Depending on the complexity and size of the data, this can cause the multiprocessing script to take longer to run than a script without multiprocessing.
- In many cases, the final step in the multiprocessing workflow is to aggregate all results together, which is an additional cost.
- Memory. When an arcpy seasoned python.exe process runs out of memory while running a geoprocessing task, it tends to thow a rather obscure and unexpected errors in tools that otherwise (when run by themselves) run just fine.
- Python Multiprocessing vs. Parallel Python:There doesn't seem to be much speed difference between the two, however:
- Parallel Python requires an extra package to be installed (Multiprocessing is included with Python)
- Parallel Python will fail if you use arcpy.addMessage() or arcpy.addError() within the parallelised function
- Parallel Python has issues if you are using tools that must be ‘Checked Out' before execution (such as Network Analyst)
- I have not been able to get Multiprocessing to work with Geocoding
- arcpy.Exists() is essential– it should be used to check every input dataset. Not just because it will catch input errors, but without it, Arc will regularly fail to do basic operations for no apparent reason. It may be that checking for existence forces a file refresh…?
- Running geoprocessing services on ArcGIS Server from ArcGIS for Desktop
- Batch jobs with ArcGIS Server:
- Large volume map generation via grid computing - ArcWeb Services and batch geocoding + routing + map generation.
- ArcGIS Server in Practice Series: Large Batch Geocoding
- ArcGIS Server includes also the ArcPy library. So it shoud be possible to run the same Python scripts as with ArcGIS for Desktop also on server. Theoretically it is possible to do that also in Linux environment: