H M < >
CMU597: Industry Project - Lecture 7

Sprint 2: Integrations Sprint

Part A: Integration Concepts | Part B: Build Integration

James Williams

Birmingham Newman University

jwilliams@staff.newman.ac.uk

3-hour session • 2 parts • 2 team tasks

Session Timeline

Part A: Integrating APIs, Libraries, and Services (90 minutes)

30 minutes: Lecture on integration patterns, error handling, testing

45 minutes: Task 1 - Plan and design integration work

15 minutes: Break

Part B: Build Integration Feature (90 minutes)

30 minutes: Implementation examples and guidance

45 minutes: Task 2 - Implement integration feature

15 minutes: Testing and troubleshooting

Learning Objectives

  • Understand different types of integrations (APIs, libraries, services)
  • Learn REST API consumption patterns
  • Implement proper error handling and retry logic
  • Build resilient integrations that handle failures gracefully
  • Test external integrations with mocks and stubs
  • Integrate third-party libraries safely
  • Work with cloud services (storage, ML inference, etc.)
  • Document integration dependencies and setup

Types of Integrations

1. REST APIs:

  • External data sources (weather, maps, payment)
  • Your own backend API (frontend → backend)
  • Third-party services (Stripe, SendGrid, Twilio)

2. Libraries & Packages:

  • npm/pip packages (React, Express, pandas, scikit-learn)
  • UI component libraries (Material-UI, Bootstrap)
  • Utility libraries (Lodash, Moment.js, Axios)

3. Cloud Services:

  • File storage (AWS S3, Google Cloud Storage)
  • Databases (Firebase, MongoDB Atlas, Supabase)
  • ML services (Google Vision API, OpenAI API)

REST API Consumption Basics

REST: REpresentational State Transfer - architectural style using HTTP methods.

HTTP Methods:

  • GET: Retrieve data (e.g., get user profile)
  • POST: Create new resource (e.g., create order)
  • PUT/PATCH: Update existing resource
  • DELETE: Remove resource
// Fetch API (browser/Node.js)
const response = await fetch('https://api.example.com/users/123');
const user = await response.json();

// Axios (more features)
const { data } = await axios.get('https://api.example.com/users/123');
console.log(data);

Example: Integrating Weather API

// OpenWeatherMap API integration
const API_KEY = process.env.WEATHER_API_KEY;
const BASE_URL = 'https://api.openweathermap.org/data/2.5';

async function getWeather(city) {
  try {
    const url = `${BASE_URL}/weather?q=${city}&appid=${API_KEY}&units=metric`;
    const response = await fetch(url);
    
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    
    const data = await response.json();
    
    return {
      temperature: data.main.temp,
      description: data.weather[0].description,
      humidity: data.main.humidity
    };
  } catch (error) {
    console.error('Weather API error:', error);
    throw error;
  }
}

// Usage
const weather = await getWeather('Birmingham');
console.log(`Temperature: ${weather.temperature}°C`);

Robust Error Handling

Common Failure Scenarios:

  • Network errors (no internet, timeout)
  • HTTP errors (404 Not Found, 500 Server Error)
  • Invalid API key or authentication failure
  • Rate limiting (too many requests)
  • Malformed response data
async function fetchWithErrorHandling(url) {
  try {
    const response = await fetch(url, { 
      timeout: 5000  // 5 second timeout
    });
    
    // Check HTTP status
    if (response.status === 429) {
      throw new Error('Rate limit exceeded. Please try again later.');
    }
    
    if (response.status === 401) {
      throw new Error('Unauthorized. Check API key.');
    }
    
    if (!response.ok) {
      throw new Error(`API error: ${response.status}`);
    }
    
    const data = await response.json();
    return data;
    
  } catch (error) {
    if (error.name === 'AbortError') {
      throw new Error('Request timeout');
    }
    throw error;
  }
}

Retry Logic for Resilience

Retry Pattern: Automatically retry failed requests after temporary failures.
async function fetchWithRetry(url, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const response = await fetch(url);
      
      if (response.ok) {
        return await response.json();
      }
      
      // Don't retry client errors (4xx)
      if (response.status >= 400 && response.status < 500) {
        throw new Error(`Client error: ${response.status}`);
      }
      
      // Retry server errors (5xx)
      if (attempt < maxRetries) {
        const delay = Math.pow(2, attempt) * 1000;  // Exponential backoff
        console.log(`Retry ${attempt}/${maxRetries} after ${delay}ms`);
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }
      
      throw new Error(`Failed after ${maxRetries} attempts`);
      
    } catch (error) {
      if (attempt === maxRetries) throw error;
    }
  }
}

API Authentication Methods

API Keys (Simple):

// In headers
fetch(url, {
  headers: {
    'X-API-Key': API_KEY
  }
});

// In query params
fetch(`${url}?api_key=${API_KEY}`);

Bearer Tokens (OAuth):

// Get token first
const token = await getAuthToken();

// Use in requests
fetch(url, {
  headers: {
    'Authorization': `Bearer ${token}`
  }
});
Security: NEVER commit API keys to Git! Use environment variables.

Integrating External Libraries

Best Practices:

  • Check Trustworthiness: npm downloads, GitHub stars, last updated
  • Review License: MIT/Apache OK, GPL may have restrictions
  • Check Dependencies: Avoid libraries with many dependencies
  • Test Before Integrating: Try in simple example first
  • Pin Versions: Use exact versions in package.json

Example: Adding Chart.js for Data Visualization

# Install library
npm install chart.js

# Import and use
import { Chart } from 'chart.js';

const chart = new Chart(ctx, {
  type: 'bar',
  data: { labels: ['A', 'B'], datasets: [{data: [10, 20]}] }
});

Example: AWS S3 File Upload

// Using AWS SDK
const AWS = require('aws-sdk');
const fs = require('fs');

// Configure SDK
const s3 = new AWS.S3({
  accessKeyId: process.env.AWS_ACCESS_KEY,
  secretAccessKey: process.env.AWS_SECRET_KEY,
  region: 'eu-west-2'
});

async function uploadFile(filePath, bucketName) {
  const fileContent = fs.readFileSync(filePath);
  
  const params = {
    Bucket: bucketName,
    Key: `uploads/${Date.now()}-${path.basename(filePath)}`,
    Body: fileContent,
    ContentType: 'image/jpeg'
  };
  
  try {
    const result = await s3.upload(params).promise();
    console.log('Upload success:', result.Location);
    return result.Location;  // Public URL
  } catch (error) {
    console.error('Upload failed:', error);
    throw error;
  }
}

// Usage
const url = await uploadFile('./photo.jpg', 'my-bucket');

Example: Google Vision API Integration

# Python example - Image classification with Google Vision
from google.cloud import vision
import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'credentials.json'

def classify_image(image_path):
    client = vision.ImageAnnotatorClient()
    
    with open(image_path, 'rb') as image_file:
        content = image_file.read()
    
    image = vision.Image(content=content)
    response = client.label_detection(image=image)
    labels = response.label_annotations
    
    if response.error.message:
        raise Exception(f'API Error: {response.error.message}')
    
    results = []
    for label in labels:
        results.append({
            'label': label.description,
            'confidence': label.score
        })
    
    return results

# Usage
labels = classify_image('dog.jpg')
for label in labels:
    print(f"{label['label']}: {label['confidence']:.2%}")

Testing External Integrations

Challenges:

  • External APIs cost money (per request)
  • APIs may be slow or unreliable
  • Hard to test error scenarios
  • Tests depend on external service

Solution: Mocks and Stubs

// Production code
async function fetchUserData(userId) {
  const response = await fetch(`https://api.example.com/users/${userId}`);
  return await response.json();
}

// Test with mock
jest.mock('node-fetch');

test('fetchUserData returns user', async () => {
  // Mock the fetch response
  fetch.mockResolvedValue({
    ok: true,
    json: async () => ({ id: 1, name: 'Tom' })
  });
  
  const user = await fetchUserData(1);
  expect(user.name).toBe('Tom');
});

Handling API Rate Limits

Rate Limit: APIs restrict number of requests per time period (e.g., 100/hour).

Strategies:

  • Cache Responses: Store results, don't re-fetch unnecessarily
  • Queue Requests: Process requests at controlled rate
  • Respect Headers: Check X-RateLimit-Remaining header
  • Implement Backoff: Wait before retrying after rate limit error
// Simple rate limiter
class RateLimiter {
  constructor(maxRequests, perSeconds) {
    this.maxRequests = maxRequests;
    this.perSeconds = perSeconds;
    this.requests = [];
  }
  
  async acquire() {
    const now = Date.now();
    this.requests = this.requests.filter(time => now - time < this.perSeconds * 1000);
    
    if (this.requests.length >= this.maxRequests) {
      const waitTime = this.perSeconds * 1000 - (now - this.requests[0]);
      await new Promise(resolve => setTimeout(resolve, waitTime));
      return this.acquire();
    }
    
    this.requests.push(now);
  }
}

Task 1: Plan and Design Integration

Instructions (Work in your project teams):

  1. Identify Integration Needs (10 min):
    • What external APIs, services, or libraries does your project need?
    • Examples: Payment (Stripe), Maps (Google Maps), Email (SendGrid), Charts (Chart.js)
  2. Select ONE Integration (5 min):
    • Choose the highest priority integration to implement today
    • Ensure it's achievable in session
  3. Research & Plan (20 min):
    • Read API documentation
    • Get API key/credentials (if needed)
    • Find code examples
    • Plan error handling approach
  4. Design Integration (10 min):
    • Sketch data flow (frontend → backend → external API)
    • Define success and error responses
    • Document in docs/integrations.md

Time: 45 minutes

Deliverable: Integration plan with API docs, examples, and design

Break Time

15 Minutes

Take a break. Next: Building your integration!

Part B: Implement Integration Feature

Part B: Implementing Your Integration

Time to code! Implement the integration you planned.

Implementation Checklist:

  • ✓ Install necessary packages/SDKs
  • ✓ Set up environment variables for secrets
  • ✓ Implement basic API call
  • ✓ Add error handling
  • ✓ Test with real requests
  • ✓ Add loading states in UI
  • ✓ Display results to user
  • ✓ Document setup in README

Example: Stripe Payment Integration

// 1. Install Stripe
// npm install stripe

// 2. Backend - Create payment intent
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

app.post('/api/create-payment', async (req, res) => {
  try {
    const { amount } = req.body;
    
    const paymentIntent = await stripe.paymentIntents.create({
      amount: amount * 100,  // Convert to cents
      currency: 'gbp',
      automatic_payment_methods: { enabled: true }
    });
    
    res.json({ clientSecret: paymentIntent.client_secret });
  } catch (error) {
    console.error('Payment error:', error);
    res.status(500).json({ error: error.message });
  }
});

// 3. Frontend - Collect payment
const { clientSecret } = await fetch('/api/create-payment', {
  method: 'POST',
  body: JSON.stringify({ amount: 50 })
}).then(r => r.json());

const result = await stripe.confirmCardPayment(clientSecret);
if (result.error) {
  alert('Payment failed: ' + result.error.message);
} else {
  alert('Payment successful!');
}

User Experience: Loading States

Always show loading state: External APIs can take seconds to respond.
// React example
function WeatherWidget({ city }) {
  const [weather, setWeather] = useState(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState(null);
  
  async function fetchWeather() {
    setLoading(true);
    setError(null);
    
    try {
      const data = await getWeatherAPI(city);
      setWeather(data);
    } catch (err) {
      setError(err.message);
    } finally {
      setLoading(false);
    }
  }
  
  if (loading) return 
Loading weather...
; if (error) return
Error: {error}
; if (!weather) return ; return
Temperature: {weather.temp}°C
; }

Caching to Reduce API Calls

// Simple in-memory cache
const cache = new Map();
const CACHE_DURATION = 5 * 60 * 1000;  // 5 minutes

async function fetchWithCache(url) {
  const cached = cache.get(url);
  
  if (cached && Date.now() - cached.timestamp < CACHE_DURATION) {
    console.log('Cache hit!');
    return cached.data;
  }
  
  console.log('Cache miss, fetching...');
  const data = await fetch(url).then(r => r.json());
  
  cache.set(url, {
    data: data,
    timestamp: Date.now()
  });
  
  return data;
}

// Usage
const weather = await fetchWithCache('https://api.weather.com/...');
// Second call within 5 minutes returns cached result

Managing API Credentials

# .env file (NOT committed to Git)
STRIPE_SECRET_KEY=sk_test_xxxxxxxxxxxxx
STRIPE_PUBLIC_KEY=pk_test_xxxxxxxxxxxxx
WEATHER_API_KEY=your_api_key_here
GOOGLE_MAPS_API_KEY=your_google_key

# .env.example (COMMITTED to Git - template for team)
STRIPE_SECRET_KEY=your_stripe_secret_key
STRIPE_PUBLIC_KEY=your_stripe_public_key
WEATHER_API_KEY=your_weather_api_key
GOOGLE_MAPS_API_KEY=your_google_maps_key
// Load in application
require('dotenv').config();

const apiKey = process.env.WEATHER_API_KEY;
if (!apiKey) {
  throw new Error('WEATHER_API_KEY not set in environment');
}
Team Setup: Each member creates .env from .env.example with their own keys

Documenting Your Integrations

docs/integrations.md Template:

# Project Integrations

## Stripe Payment Processing

**Purpose:** Accept payments for bookings

**Setup:**
1. Sign up at stripe.com, get API keys
2. Add keys to `.env`:
   ```
   STRIPE_SECRET_KEY=sk_test_...
   STRIPE_PUBLIC_KEY=pk_test_...
   ```
3. Install: `npm install stripe`

**Usage:**
- Backend: `/api/create-payment` endpoint
- Frontend: Stripe Elements for card collection

**Rate Limits:** None in test mode
**Cost:** Free in test mode
**Documentation:** https://stripe.com/docs

Task 2: Build Integration Feature

Instructions (Work in your project teams):

  1. Setup (10 min):
    • Install required packages
    • Create .env file with API keys
    • Share .env.example with team
  2. Implement Backend (if needed) (15 min):
    • Create API endpoint that calls external service
    • Add error handling and validation
  3. Implement Frontend (15 min):
    • Add UI to trigger integration
    • Show loading state
    • Display results or errors
  4. Test & Document (5 min):
    • Test with real API calls
    • Document setup in docs/integrations.md

Time: 45 minutes

Deliverable: Working integration with error handling and documentation

Lecture 7 Summary

  • Integrations connect your system to external APIs, libraries, and services
  • REST APIs use HTTP methods (GET, POST, PUT, DELETE)
  • Always implement error handling for network failures and API errors
  • Retry logic with exponential backoff improves resilience
  • Use environment variables for API keys and secrets
  • Mock external APIs in tests to avoid costs and dependencies
  • Implement loading states for better user experience
  • Cache responses to reduce API calls and improve performance
  • Document all integrations for team reference

Next Lecture:

Sprint 3 - Refinement & Quality
Refactoring, code quality, testing strategies, and cleaning up technical debt

Before Next Lecture

  1. Complete Integration:
    • Finish implementation if not done
    • Test thoroughly with edge cases
    • Handle all error scenarios
  2. Documentation:
    • Update docs/integrations.md
    • Add setup instructions to README
    • Document API costs/rate limits
  3. Team Sharing:
    • Ensure all team members can run integration
    • Share .env.example file
  4. Submit to Moodle:
    • Link to integration PR
    • Screenshot of working integration
    • docs/integrations.md