Note: this page is only a draft, but this project is hosted on a public repository where anyone can contribute. Learn how to contribute in less than a minute.

Big data

Big data is data sets that are so voluminous and complex that traditional data-processing application software are inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source. | Source: wikipedia.

Table of contents

• Introduction
• Technology
  • Esri products
  • Other tools an projects
• Training
  • Documentation
  • Videos
  • Slides
• Awesome projects and cool demos
• Support and Community
  • People you should know
• News
• Additional resources
  • Localized resources
  • Other related list of resources
• Partners and startups
• Contributions

Introduction

Big data use to be related IoT in the platform, but that's not always the case, for example, some times it is also related with the management of vast amounts of images.. That's why we have created a separated page for ArcGIS and IoT.

Technology

Tools and products to work with big datasets.

Esri products

Based on your needs There are a set of technologies in the ArcGIS platform you might want to know if you are interested in this topic:

• Storage, search, querying & updating data -> Spatiotemporal Big Data Store: we offer the spatiotemporal big data store to enable archival of high volume observation data, sustains high velocity write throughput, and can run across multiple machines (nodes). But ArcGIS is also compatible with other storage technologies like Apache Hadoop.

• Capture, transfer, sharing -> GeoEvent Server: enables you to work with real-time mapping and analytics. It works with: MongDB, Hadoop, RabbitMQ, ActiveMQ, etc.

• Data analysis:

  • Vector Data (Real-Time) -> GeoEvent Server
  • Vector Data (Batch) -> GeoAnalytics performs analysis with large volume of data
  • Raster Data (Batch) -> ArcGIS Image Server provides serving, processing, analysis, and extracting value from massive collections of imagery, rasters, and remotely sensed data.

• Visualization -> JavaScript API and SDK Runtimes, (learn more about the developer tecnologies).

Note: ArcGIS API for Python: the arcgis.geoanalytics module provides submodules, data types, classes and functions to process your big data using an ArcGIS GeoAnalytics Server (more info)

Other tools an projects

• GIS Tools for Hadoop
• Spark projects from Mansour Raad
  • spark-pip: Spark job to perform massive Point in Polygon (PiP) operations
  • spark-gdb: A library for parsing and querying an Esri File Geodatabase with Apache Spark.
  • Spark SQL DBF Library: Spark SQL DBF Library
  • SparkProject: Using Apache Spark in an ArcMap Toolbox
  • dbscan-spark: DBSCAN implementation using Apache Spark
  • hdfs-geohex: (Web)Mapping Elephants with Sparks
  • Spark, Cassandra, Tessellation and ArcGIS](Spark, Cassandra, Tessellation and ArcGIS
  • ...

Training

• An Introduction to Big Data (by Esri)
• GIS Tools for Hadoop for Beginners
• New Spatial Aggregation Tutorial for GIS Tools for Hadoop
• Tutorial: Run a GeoAnalytics tool
• From the Big Data Blog at GeoNet
  • Aggregating layers using GeoAnalytics
• How to read and write layers in pyspark using the Run Python Script task through the REST API in ArcGIS Enterprise

Note**: Check for specific training about GeoEvent, GeoAnalytics, Spatiotemporal Big Data Store and ArcGIS Image Server on their respective pages.

Documentation

• GeoEvent Server input connectors (Kakfa, RabbitMQ, ActiveMQ, ...)
• Get started with big data file shares in GeoAnalytics Server (ArcGIS Enterprise)

Videos

On video.esri.com and Esri Events you will find many technical talks. Or check Esri Geodev channel for short videos.

You might also find interesting videos at Esri Industries, ArcGIS, EsriTV, Esri R&D Center etc

Slides

More slides in proceedings.esri.com or check on GeoNet.

Awesome projects and cool demos

• GeoAnalytics at DevSummit Plenary 2019

• Raster Analytics at DevSummit Plenary 2019

• Spatial Big Data Solution: Airbnb Los Angeles

• Taxi Cab Terrain: Millions of Cab Rides Over One Year Paint a Portrait of New York City
• Big Data Analytics on Citi Bike

Note: Check also the cool demos and projects at GeoEvent, GeoAnalytics, Spatiotemporal Big Data Store and ArcGIS Image Server pages.

Find more BigData use cases here.

Support and Community

If needed, remember you can always check the Technical Support Website for developer products and the Technical Support Blog at GeoNet to find additional resources.

Community channels:

• Big Data GeoNet Group
  • Answered questions
• GIS Tools for Hadoop issues

Is there any category on ArcGIS Ideas on GeoNet for this topic?

Vote/share the ideas about Big Data @ [ArcGIS Ideas on GeoNet](https://community.esri.com/search.jspa?q=Big Data&place=%2Fplaces%2F478947&depth=ALL)

People you should know

Please find the ArcGIS Experts on this topic using this tool:

They don't work on the support team and some may even not work at Esri, but they are opening some projects and sharing insights related to this technology so we recommend you to follow them.

You can also explore the experts in the Awesome ArcGIS organization in ArcGIS Online.

News

• Big Data at esri.com search engine
• Blog Posts about Big Data inside Places at GeoNet

Additional resources

Probably not all the resources are in this list, please use the ArcGIS Search tool looking for: "big data".

Localized resources

• Taller de GeoaAalytics de un día
• IoT y Big Data: los mejores aliados
• Cómo adoptar una estrategia de éxito en Big Data y Artificial Intelligence con AWS - CEsri17

Other related list of resources

• Performance:
  • ArcGIS performance
  • Learn more about how the ArcGIS API for Javascript is using WebGL, the GPU, WebAssembly, HTTP2, Protocol Buffers (pbfs), a client side geometry engine, etc. to deliver a high performance experience | JSAPI performance
  • ArcGIS and High Performance Computing (HPC)
• Imagery data
• Other emerging technologies related:
  • Artificial intelligence
  • IoT-Real Time

Partners and startups

Organizations and the type of relation with this product:

Is there any partner or startup you can had with expertise with this product?, learn how to contribute in less than a minute

Contributions

Feel free to improve/extend this resource page using this template (Contribution Guide).