API Documentation - Dubai Real Estate

🎯 Overview

A comprehensive platform for analyzing and accessing Dubai real estate market data via REST APIs. This system provides property transaction data and powerful APIs for developers and businesses.

API Playground: http://localhost:8080/playground
API Documentation: http://localhost:8080/documentation
API Base URL: http://localhost:3000/api/v1

Key Features

RESTful API: 20+ endpoints for property transactions, valuations, analytics, and market insights
Interactive Playground: Test all APIs with live code generation in 4 languages (Shell, JavaScript, Python, PHP)
Automated Data Collection: Intelligent duplicate detection and progressive data saving
Market Intelligence: Location trends, rental yield calculations, price movements, market segmentation, and seasonal patterns
PostgreSQL Database: Robust data storage with optimized queries and indexing
Real-time Data: Access to 1,300+ property transactions across 170+ locations

What You Can Do

🔍 Search and filter property transactions with 10+ filters
💰 Get instant property valuations with comparable sales
📈 Track price trends across 170+ locations
🏢 Analyze 420+ buildings and their performance
📊 Compare multiple areas side-by-side
🗺️ Identify market hotspots and trending areas
🎮 Test APIs interactively in the playground
💻 Integrate with your application using code examples

🚀 Getting Started

1. Access the API Playground

Navigate to http://localhost:8080/playground in your web browser. No login required - the playground is publicly accessible.

💡 Quick Start: The playground provides an interactive interface to test all API endpoints. You can generate code examples in multiple languages and execute requests directly from your browser.

2. Load Sample Data

Before testing endpoints, you need to load data into the system:

In the playground, expand "Data Management" in the left sidebar
Click "Load Sample Data" (if available) or use the API endpoint directly
Alternatively, use the data collection endpoint to load fresh property data

3. Test an API Endpoint

Try a quick property valuation:

Click "Valuations" > "Quick Valuation" in the sidebar
Fill in the parameters (area, property type, bedrooms, size)
Click "Try It!" to execute the request
View the response and copy the generated code

4. Integrate into Your Application

Use the generated code examples to integrate the API into your application. The playground supports Shell (cURL), JavaScript, Python, and PHP code generation.

📊 Data Collection

The platform includes a sophisticated data collection system that processes property transaction data.

Key Capabilities

Persistent Sessions: Maintains session state for reliable data collection
Progressive Saving: Saves data incrementally during collection, not just at the end
Intelligent Deduplication: Checks existing transactions to avoid duplicate entries
Dual Format Parser: Handles both SALE (table layout) and RENTAL (card layout) data structures
Comprehensive Logging: Tracks every data collection operation with duration, results, and errors
Error Recovery: Continues data collection even if individual operations fail

How It Works

1. Initialize data collection system
2. For each location (173 locations):
   - Collect SALE transactions (table format)
   - Collect RENTAL transactions (card format)
   - Parse property details (price, size, bedrooms, etc.)
   - Check for duplicates in database
   - Save new transactions
   - Log operation results
3. Complete collection and display summary

Data Fields

Field	Description	Example
location	Area name	"Dubai Marina"
building_name	Building/project name	"Marina Heights Tower"
property_type	Type of property	"apartment", "villa", "townhouse"
bedrooms	Number of bedrooms	0 (studio), 1, 2, 3, 4, 5+
size_sqft	Size in square feet	1200
price	Transaction price in AED	2500000
deal_type	Sale or rental	"sale", "rental"
transaction_date	Date of transaction	2026-01-15
developer_name	Property developer	"Emaar Properties"

Performance

Typical Data Collection Run:
• Duration: 30-45 minutes for all locations
• Pages processed: ~300-400 pages
• Properties found: ~13,000-15,000
• New transactions saved: ~1,000-2,000
• Duplicates skipped: ~12,000
• Success rate: 95%+

🗄️ Data Structure

The system uses PostgreSQL with two main tables:

1. Transactions Table

CREATE TABLE transactions (
  id SERIAL PRIMARY KEY,
  location VARCHAR(255),
  building_name VARCHAR(255),
  property_type VARCHAR(100),
  bedrooms INTEGER,
  size_sqft DECIMAL(10,2),
  price DECIMAL(15,2),
  deal_type VARCHAR(20),  -- 'sale' or 'rental'
  transaction_date DATE,
  developer_name VARCHAR(255),
  created_at TIMESTAMP DEFAULT NOW()
);

-- Indexes for performance
CREATE INDEX idx_location ON transactions(location);
CREATE INDEX idx_building ON transactions(building_name);
CREATE INDEX idx_date ON transactions(transaction_date DESC);
CREATE INDEX idx_price ON transactions(price);
CREATE INDEX idx_deal_type ON transactions(deal_type);

2. Data Collection Log Table

CREATE TABLE data_collection_log (
  id SERIAL PRIMARY KEY,
  collection_type VARCHAR(50) NOT NULL,
  location VARCHAR(255),
  mode VARCHAR(20),  -- 'sale' or 'rental'
  page_number INTEGER,
  properties_found INTEGER DEFAULT 0,
  properties_saved INTEGER DEFAULT 0,
  duplicates_skipped INTEGER DEFAULT 0,
  errors TEXT,
  started_at TIMESTAMP DEFAULT NOW(),
  completed_at TIMESTAMP,
  duration_seconds INTEGER,
  status VARCHAR(20) DEFAULT 'running'
);

CREATE INDEX idx_data_collection_log_status ON data_collection_log(status);
CREATE INDEX idx_data_collection_log_started ON data_collection_log(started_at DESC);

Current Database Stats

Total Transactions: 1,315
Unique Locations: 173
Unique Buildings: 421
Date Range: Dec 20, 2025 - Jan 19, 2026
Price Range: AED 1 - AED 52,976,739
Average Price: AED 1,750,378
Property Types: apartment (869), villa (52)
Deal Types: sale (869), rental (446)

🎮 API Playground

The platform includes an interactive API playground inspired by Unipile's developer portal for testing endpoints.

Features

3-Column Layout: Sidebar navigation, endpoint details center, code examples right
Live Code Generation: Auto-updates as you configure parameters
Multiple Languages: Shell (cURL), JavaScript (Fetch), Python (Requests), PHP
Try It! Button: Execute API calls directly from the browser
Response Viewer: See formatted JSON responses with status codes
Copy to Clipboard: One-click copy of code examples

How to Use

Navigate to http://localhost:8080/playground
Browse endpoints in the left sidebar (organized by category)
Click an endpoint to load it
Fill in parameters in the center panel
Choose your programming language (Shell, JavaScript, Python, PHP)
Click "Try It!" to execute the request
View the response below
Click "Copy" to copy the code example

Note: The playground connects to the business API endpoints running on port 3000. Make sure the API server is running before testing endpoints.

⚙️ Technical Architecture

Technology Stack

Layer	Technology	Purpose
Backend	Node.js + Express.js	Web server and API endpoints
Database	PostgreSQL 14	Transaction and log storage
Data Collection	Playwright (Chromium)	Browser automation for data processing
Charts (Standard)	Chart.js 4.4.1	Bar, line, donut, radar charts
Charts (Advanced)	D3.js v7	Bubble, treemap, heatmap, scatter, box plot, quadrant
Authentication	express-session	Session-based auth with cookies
Date Handling	chartjs-adapter-date-fns	Time-based chart axes

Project Structure

dxbinteractstudy/
├── src/
│   ├── server.js              # Main API server (port 3000)
│   ├── apiV1.js               # Business API endpoints
│   ├── dataCollector.js       # Data collection logic
│   ├── database.js            # PostgreSQL connection pool
│   └── dataStore.js           # In-memory data cache
├── webapp/
│   ├── server.js              # Webapp server (port 8080)
│   └── public/
│       ├── playground.html    # API playground interface
│       ├── playground.css     # Shared styles
│       ├── playground.js       # Playground logic
│       └── documentation.html # This documentation
├── dataCollector.js           # Data collection CLI entry point
├── docs/                      # Markdown documentation
├── .env                       # Environment variables
└── package.json               # Dependencies

Environment Variables

# PostgreSQL Connection
PGUSER=your_username
PGHOST=localhost
PGDATABASE=real_estate_db
PGPASSWORD=your_password
PGPORT=5432

# Webapp Configuration
WEBAPP_PORT=8080
API_PORT=3000

# Session Security (optional)
API_KEY_SECRET=your_session_secret

Port Allocation

Port 8080: Webapp server (playground and documentation)
Port 3000: Business API server (REST endpoints)
Port 5432: PostgreSQL database

Performance Optimizations

Database indexes on location, building, date, price, deal_type
Parallel chart rendering using Promise.all()
Progressive data saves (doesn't wait till end)
Duplicate detection before INSERT (prevents duplicates)
Session-based auth (no token overhead)
Static file caching for CSS/JS assets

Security Features

Public API access (no authentication required for playground and documentation)
SQL parameterized queries (injection protection)
Environment variable secrets (not hardcoded)
CORS enabled for cross-origin requests
Input validation on API endpoints

📊 Dubai Real Estate Analytics Platform

📑 Table of Contents