Sort (Data Management) on a Basic License with ArcPy

Glen Bambrick

July 14, 2024
12:02 pm
No Comments

Introduction

The Sort (Data Management) tool has limited functionality on a Basic and Standard licence, you can only sort by one field and cannot use a spatial sort. We saw in this blog post how we can use the Export Features (Conversion) tool to sort a feature class or table by multiple fields, unfortunately that Export tool does not sort spatially, and that’s where Final Draft Mapping comes in to help. In this post we are going to re-create the Sort tool for use with a Basic license. You can check out the Esri documentation for the Sort tool here.

ArcPy Training Spotlight

ArcPy for Data Management and Geoprocessing with ArcGIS Pro

Leverage ArcPy for geospatial data management workflows within ArcGIS Pro. Learn the fundamentals of utilising ArcGIS Pro geoprocessing tools with ArcPy for data management, conversions, and analysis. This course is packed with amazing content that will help you discover the power of automating tasks within the ArcGIS Pro environment. Take your ArcPy skills from beginner to snake charmer. A little code goes a long way, a little ArcPy provides you with an in-demand skill. Sign up now for our highly rated course.

Syntax

The syntax for the Sort tool is shown below.

				
					arcpy.management.Sort(
    in_dataset, 
    out_dataset, 
    sort_field,
    {spatial_sort_method}
)

Import the ArcPy module

ArcPy is a Python site package that enables automation, analysis, and management of geographic data within the ArcGIS software environment.

				
					import arcpy

Define the parameters

As per the Sort tool syntax we require four possible user inputs; the in_dataset is the feature class you wish to perform the sort on, we won’t bother with tables as you can use the Export Table (Conversion) tool to sort those bad boys; out_dataset will be our sorted output dataset; and sort_field is for the field or fields you wish to sort by; spatial_sort_method, if the Shape field has been supplied (we will supply as default) you need to supply a Spatial Sort Method, and if it hasn’t been used then we’re pretty mush just using the Export Features tool!

Just to note, we will only be covering four out of the five spatial sorts available; Upper Right (default), Lower Right, Upper Left, and Lower Left. The Peano Curve algorithm will be omitted, purely because I need to provide more brain power to it at another stage.

				
					## the features to sort
in_dataset = arcpy.GetParameterAsText(0)

## the output feature class or table to create
out_dataset = arcpy.GetParameterAsText(1)

## field whose values will be used to reorder the input records
sort_field = arcpy.GetParameterAsText(2)

## spatial sort method: UR, LR, UL, LL
spatial_sort_method = arcpy.GetParameterAsText(3)

Required Python object for the tool

We have a few required objects to help us through the workflow as detailed in the code below.

				
					## srs id of the in_dataset so we can assign the output the same
srs_id = arcpy.Describe(in_dataset).spatialReference.factoryCode

## all field names from the input dataset
all_fields = [fld.name for fld in arcpy.ListFields(in_dataset) if fld.type not in ("GlobalID", "Guid", "OID", "Geometry") and not fld.name.startswith("Shape_")]

## we will need to the geometry to add into the output.
all_fields.append("SHAPE@")

## add oid field to beginning of the all_fields list
oid_fld = [fld.name for fld in arcpy.ListFields(in_dataset) if fld.type == "OID"][0]
all_fields.insert(0, oid_fld)

## this list will hold the names of the fields to sort by and order by
sort_fields =[entry.split(" ") for entry in sort_field.replace("'", "").split(";")]

## required to create correct output geometry type
shape_type = arcpy.Describe(in_dataset).shapeType

Get all input records

Ok, let’s get into it! We need all the records from the in_dataset as a list (of attributes/geometry) in a list.

				
					## this will be a list of list.
all_records = [list(row) for row in arcpy.da.SearchCursor(in_dataset, all_fields)]

Which way are we sorting?

Next, we handle if a spatial sort method has been supplied. Based on the method desired by the user, we get the bounding box coordinates for that selection at each record. For example, if Lower Left (LL) is selected, then we get the lower-left (Xmin, YMin) coordinates of the bounding extent. The bounding extent for a point will always be the centroid. For each possible spatial sort we either reverse or do not reverse the values when using the Python sorted() method. Once we have applied our sorts we remove the X and Y coordinates from our records.

				
					## if a spatial sort method is being applied
if spatial_sort_method != "None":
    ## and the sort method us Upper Right
    if spatial_sort_method == "UR":
        ## get the Upper Right of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMax, record[-1].extent.YMax] for record in all_records]
        x_reverse = True
        y_reverse = True
    ## if the sort method is Lower Right
    elif spatial_sort_method == "LR":
        ## get the Lower Right of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMax, record[-1].extent.YMin] for record in all_records]
        x_reverse = True
        y_reverse = False
    ## if the sort method is Upper Left
    elif spatial_sort_method == "UL":
        ## get the Upper Left of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMin, record[-1].extent.YMax] for record in all_records]
        x_reverse = False
        y_reverse = True
    ## if the sort method is Lower Left
    elif spatial_sort_method == "LL":
        ## get the Lower Left of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMin, record[-1].extent.YMin] for record in all_records]
        x_reverse = False
        y_reverse = False
    ## sort by x ascending
    all_records = sorted(all_records, key=lambda row:row[-2], reverse=x_reverse)
    ## sort by y descending
    all_records = sorted(all_records, key=lambda row:row[-1], reverse=x_reverse)
    ## we're finished with the X and Y so remove from our records
    all_records = [record[0:-2] for record in all_records]

Create a temporary feature class and define schema

We make use of the memory workspace and create a feature class based on the shape type of the in_dataset. The in_dataset is used as a template. The ORIG_FID field is added as this will form part or the output.

				
					## temporary feature class in memory workspace using in_dataseta as a template
temp_fc = arcpy.CreateFeatureclass_management("in_memory", "temp_fc", shape_type,
    in_dataset, "SAME_AS_TEMPLATE", "SAME_AS_TEMPLATE", srs_id)
## add ORIG_FID field
arcpy.management.AddField(temp_fc, "ORIG_FID", "LONG")

Insert records into memory feature class

Now we insert our spatially sorted records into our memory feature class. If the Shape field was not used and there was no spatial_sort method set, we are really just inserting all our records back in as is. There might be an alternative way to achieve this, but this method allows for the ORIG_FID to be added to the output.

				
					## remove OID field name and replace with ORIG_FID
all_fields[0] = "ORIG_FID"
## use an insert cursor on the temp_fc
with arcpy.da.InsertCursor(temp_fc, all_fields) as i_cursor:
    ## for each record in all_records
    for record in all_records:
        i_cursor.insertRow(record)

Export to disk

We can now write our temporary feature class from the memory workspace to disk.

				
					## remove Shape entry from the sort_fields if in there
sort_field = [x for x in sort_fields if "shape" not in str(x).lower()]
arcpy.conversion.ExportFeatures(temp_fc, out_dataset, sort_field=sort_field)

Cleanup the Memory Workspace

				
					## clean-up memeory workspace
arcpy.management.Delete(temp_fc)

ArcPy Training Spotlight

Mastering ArcGIS Pro ArcPy Search, Insert, & Update Cursors

Unlock the full potential of ArcPy Cursors with our intensive and in-depth course. Designed for GIS professionals, analysts, and enthusiasts, this course delves into the intricacies of Search, Insert, and Update Cursors in the ArcPy library, empowering you to manipulate and manage spatial data with precision and efficiency.

ArcPy Cursors enable you to streamline your GIS workflows, automate repetitive tasks, and witness the full potential of spatial data manipulation in ArcGIS Pro.

Create the tool in ArcGIS Pro

Save your script and open ArcGIS Pro. Right-click on your toolbox/toolset of choice and select New > Script. The New Script window will appear. In the General tab set Name to Sort, Label to Sort (Basic), and the Description to Sort with a Basic License.

In the Parameters tab set as per below. Set the Output Dataset parameter Direction to Output. Set the Dependency for the Sort Field(s) to in_dataset and a Default of SHAPE ASCENDING. Set the Default for Spatial Sort Method to UR.

The Value Table for the Sort Field(s) parameter.

And the Filter for the Sort Field(s)

And the Value List Filter for Spatial Sort Method.

In the Execution tab, click the folder icon in the top-right corner and add your saved Python script.

Click into the Validation tab and update the updateParameters and updateMessages as per below.

				
					def updateParameters(self):
    # Modify parameter values and properties.
    # This gets called each time a parameter is modified, before 
    # standard validation.
    field_list = [entry.split(" ")[0].lower() for entry in self.params[2].valueAsText.replace("'", "").split(";")]
    if "shape" not in field_list:
        self.params[3].value = "None"          
    return
    
def updateMessages(self):
    # Customize messages for the parameters.
    # This gets called after standard validation.
    field_list = [entry.split(" ")[0].lower() for entry in self.params[2].valueAsText.replace("'", "").split(";")]
    if "shape" in field_list and self.params[3].value == "None":
        self.params[3].setErrorMessage("If Shape field selected, Spatial Sort Method cannot be None")
    else:
        self.params[3].clearMessage()             
    return

Click OK and go test out the tool.

You can download the tool and other Advanced tools with a Basic license over on this page.

All the code in one place

You can find the entire code workflow below with links to important components in the documentation that were used.

				
					import arcpy

################################################################################
## Esri Documentation
##  https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/sort.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/functions/getparameterastext.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/functions/describe.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/functions/listfields.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/data-access/searchcursor-class.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/data-access/insertcursor-class.htm
##  https://pro.arcgis.com/en/pro-app/latest/arcpy/classes/extent.htm
##  https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/create-feature-class.htm
##  https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/add-field.htm
##  https://pro.arcgis.com/en/pro-app/latest/tool-reference/conversion/export-features.htm
##  https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/delete.htm
##
## Notes:
##  Will only work if sorting spatially is the first or last entry in the
##  sort_field parameter, and will perform the spatial sort first and then the
##  attribute sorts.
##
## ArcGIS Pro Version 3.1.0
##
################################################################################

################################################################################
## USER INPUTS
## the features to sort
in_dataset = arcpy.GetParameterAsText(0)
## the output feature class or table to create
out_dataset = arcpy.GetParameterAsText(1)
## field whose values will be used to reorder the input records
sort_field = arcpy.GetParameterAsText(2)
## spatial sort method: UR, LR, UL, LL
spatial_sort_method = arcpy.GetParameterAsText(3)

################################################################################
## REQUIRED OBJECTS

## srs id of the in_dataset so we can assign the output the same
srs_id = arcpy.Describe(in_dataset).spatialReference.factoryCode
## all field names from the input dataset
all_fields = [fld.name for fld in arcpy.ListFields(in_dataset) if fld.type not in ("GlobalID", "Guid", "OID", "Geometry") and not fld.name.startswith("Shape_")]
## we will need to the geometry to add into the output.
all_fields.append("SHAPE@")
## add oid field to beginning of the all_fields list
oid_fld = [fld.name for fld in arcpy.ListFields(in_dataset) if fld.type == "OID"][0]
all_fields.insert(0, oid_fld)
## this list will hold the names of the fields to sort by and order by
sort_fields =[entry.split(" ") for entry in sort_field.replace("'", "").split(";")]
## required to create correct output geometry type
shape_type = arcpy.Describe(in_dataset).shapeType

################################################################################
## GET ALL RECORDS

## this will be a list of list.
all_records = [list(row) for row in arcpy.da.SearchCursor(in_dataset, all_fields)]

################################################################################
## SORT SPATIALLY IF REQUIRED

## if a spatial sort method is being applied
if spatial_sort_method != "None":
    ## and the sort method us Upper Right
    if spatial_sort_method == "UR":
        ## get the Upper Right of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMax, record[-1].extent.YMax] for record in all_records]
        x_reverse = True
        y_reverse = True
    ## if the sort method is Lower Right
    elif spatial_sort_method == "LR":
        ## get the Lower Right of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMax, record[-1].extent.YMin] for record in all_records]
        x_reverse = True
        y_reverse = False
    ## if the sort method is Upper Left
    elif spatial_sort_method == "UL":
        ## get the Upper Left of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMin, record[-1].extent.YMax] for record in all_records]
        x_reverse = False
        y_reverse = True
    ## if the sort method is Lower Left
    elif spatial_sort_method == "LL":
        ## get the Lower Left of the bounding box for all features
        ## and add as X and Y separately to the each record
        all_records = [record+[record[-1].extent.XMin, record[-1].extent.YMin] for record in all_records]
        x_reverse = False
        y_reverse = False
    ## sort by x ascending
    all_records = sorted(all_records, key=lambda row:row[-2], reverse=x_reverse)
    ## sort by y descending
    all_records = sorted(all_records, key=lambda row:row[-1], reverse=y_reverse)
    ## we're finished with the X and Y so remove from our records
    all_records = [record[0:-2] for record in all_records]
    
################################################################################
## CREATE OUPUT FEATURE CLASS IN MEMORY ########################################

## temporary feature class in memory workspace using in_dataseta as a template
temp_fc = arcpy.management.CreateFeatureclass("in_memory", "temp_fc", shape_type,
    in_dataset, "SAME_AS_TEMPLATE", "SAME_AS_TEMPLATE", srs_id)
## add ORIG_FID field
arcpy.management.AddField(temp_fc, "ORIG_FID", "LONG")

################################################################################
## INSERT RECORDS ##############################################################

## remove OID field name and replace with ORIG_FID
all_fields[0] = "ORIG_FID"
## use an insert cursor on the temp_fc
with arcpy.da.InsertCursor(temp_fc, all_fields) as i_cursor:
    ## for each record in all_records
    for record in all_records:
        i_cursor.insertRow(record)
        
################################################################################
## WRITE TO DISK ###############################################################

## remove Shape entry from the sort_fields if in there
sort_field = [x for x in sort_fields if "shape" not in str(x).lower()]
arcpy.conversion.ExportFeatures(temp_fc, out_dataset, sort_field=sort_field)

################################################################################
## CLEAN-UP MEMORY WORKSPACE ###################################################

## clean-up memeory workspace
arcpy.management.Delete(temp_fc)
################################################################################