🚀 Making HTTP Requests in Bright Data IDE

1️⃣ request() – Direct HTTP Request

Use request() when you want to fetch data directly from an API without browser interaction.

Simple GET Request

let res = request('http://www.example.com');

POST Request with Headers

let res = request({
  url: 'http://www.example.com',
  method: 'POST',
  headers: {
    'Content-type': 'application/json; charset=utf-8'
  },
  body: { hello: 'world' }
});

Pagination Example

const itemsPerPage = 20;
let currentPage = 1;
let totalPages = Infinity;

while (currentPage <= totalPages) {
  const targetUrl =
    `https://open-pro.dict.naver.com/ivo-data/dict/dictList` +
    `?currentPage=${currentPage}` +
    `&pageSize=${itemsPerPage}` +
    `&dictType=0&orderType=0&themeCode=`;

  const response = await request({
    url: targetUrl,
    method: 'GET',
    headers: {
      'Accept': 'application/json',
      'User-Agent': 'Mozilla/5.0'
    }
  });

  if (response.statusCode !== 200) {
    console.error(`Page ${currentPage} failed`);
    currentPage++;
    continue;
  }

  const data = JSON.parse(response.body);

  if (!data?.data?.items?.length) {
    console.log('No more dictionaries found');
    break;
  }

  if (currentPage === 1 && data?.data?.totalPage) {
    totalPages = data.data.totalPage;
  }

  currentPage++;
}

2️⃣ tag_request() – Tagged API Request

tag_request() allows you to fire an API request and retrieve the response later inside parser code.

Basic Usage

tag_request('req1', 'http://www.example.com');
let response = wait_for_parser_value('req1');

Real-World Product API Example

const current_sku = location.href.split('/').pop().split('?')[0];
const apiUrl =
  `https://www.loewe.com.cn/rest/cn/V1/applet/product/${current_sku}`;

const tagName = 'api_req_single';

tag_request(tagName, {
  url: apiUrl,
  method: 'GET',
  headers: {
    'accept': 'application/json, text/plain, */*',
    'content-type': 'application/json'
  }
});

try {
  wait_for_parser_value(tagName, 10000);
} catch (e) {
  console.log('Timeout waiting for tag_request');
}

let api_response = null;
if (parser[tagName]?.value) {
  api_response =
    typeof parser[tagName].value === 'string'
      ? JSON.parse(parser[tagName].value)
      : parser[tagName].value;
}

3️⃣ tag_response() – Capture Browser Network Responses

Use this when an API call is triggered automatically during page load.

tag_response('teams', /\/api\/teams/);
navigate('https://example.com/sports');

const teams = parse().teams;

Advanced Matching

tag_response('resp', (req, res) => {
  if (req.url.includes('/api/')) {
    let response_body = res.body;
  }
});

4️⃣ tag_all_responses() – Capture Multiple API Calls

Best suited for pagination APIs or repeated background requests.

tag_all_responses('profiles', /\/api\/profile/);
navigate('https://example.com/sports');

let profiles = wait_for_parser_value('profiles');

// parser code
return { profiles };