We’ve been running the Jobs Hub data collection service for close to a year now, and so far, it’s the only software we support that’s not open source. As of today, that is changing.

You can now run your very own version of our job board data collector for free by installing and running our open source job collection project. Here’s a little more about how the project works:

Why?

When I started work on JobApis, my goal was to standardize data from job boards. At some point, it became obvious that I’d need to collect this data from job boards and store it temporarily in a database in order to access it, analyze it, and learn more about the jobs being posted. The JobApis Collector is simply a wrapper around JobsMulti that allows you to dump jobs in Algolia as you collect them.

How it works

Unfortunately, there’s not an easy way to get every job from every job board supported by JobsMulti, so the approach this project takes is to use a list of search terms to search each job board. The terms can include a location and keyword as well as timestamps for when they were created, collected, and requested for collection. These objects are stored in Algolia:

1
2
3
4
5
6
7
8

{
  "keyword": "Retail Sales Associate",
  "location": "Orlando, FL",
  "created_at": 1501081885,
  "last_collection_requested_at": 1511550912,
  "last_collection_completed_at": 1511550923,
  "objectID": "354994470"
}

Each time the collector runs, it gets the term that was least recently requested for collection and queues up a request to collect it. Each collection run uses JobsMulti to get all of the latest jobs from all the jobs boards supported and saves each to Algolia. The jobs use our standard job object, and look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

{
      "alternateName": null,
      "baseSalary": null,
      "company": "ALDI",
      "datePosted": "2017-11-23",
      "description": "Store <b>Manager</b> Trainee – <b>Retail</b> Management (Customer Service) As a <b>Manager</b> Trainee at ALDI, you’ll never experience the same day twice....",
      "educationRequirements": null,
      "employmentType": null,
      "endDate": null,
      "experienceRequirements": null,
      "hiringOrganization": {
        "address": {
          "addressCountry": null,
          "addressLocality": "Orlando",
          "addressRegion": "FL",
          "postalCode": "Orlando, FL",
          "postOfficeBoxNumber": null,
          "streetAddress": null
        },
        "email": null,
        "logo": null,
        "telephone": null,
        "alternateName": null,
        "description": null,
        "name": "ALDI",
        "url": null
      },
      "incentiveCompensation": null,
      "industry": null,
      "javascriptAction": null,
      "javascriptFunction": null,
      "jobBenefits": null,
      "jobLocation": {
        "address": {
          "addressCountry": null,
          "addressLocality": "Orlando",
          "addressRegion": "FL",
          "postalCode": "Orlando, FL",
          "postOfficeBoxNumber": null,
          "streetAddress": null
        },
        "geo": {
          "latitude": null,
          "longitude": null,
          "alternateName": null,
          "description": null,
          "name": null,
          "url": null
        },
        "telephone": null,
        "alternateName": null,
        "description": null,
        "name": null,
        "url": null
      },
      "location": "Orlando, FL",
      "maximumSalary": null,
      "minimumSalary": null,
      "name": "Manager Trainee",
      "occupationalCategory": null,
      "qualifications": null,
      "query": "Retail Sales Manager",
      "responsibilities": null,
      "salaryCurrency": null,
      "skills": null,
      "source": "Indeed",
      "sourceId": "c35c3c7a5d61857e",
      "specialCommitments": null,
      "startDate": null,
      "title": "Manager Trainee",
      "url": "http://www.indeed.com/viewjob?jk=c35c3c7a5d61857e&qd=0YhWF1ggXeR_D_AC4ekfOruTBpHhozBGDIfLHBLnubD-2-W0u32WFrU39zSxqydN3J-4nOSXPQlGVSMv10v9koBwCaytx9cwv0CF6SqbAQokt0LCVy_rQinuMfW47e_m&indpubnum=3806336598146294&atk=1bvvn25klav8vcas",
      "workHours": null,
      "_terms": [
        "354994470"
      ],
      "lastObservedAt": 1511819124,
      "firstObservedAt": 1511819124,
      "objectID": "49116121"
    }

The advantage to using Algolia is that the jobs can be very quickly accessed. You can configure your own indexes to make sure the data fields you need to search are front and center. The downside to Algolia is the cost as their minimum package price is around $60 per month.

Archiving jobs

Since most users don’t need jobs that are stale and I wanted to minimize my Algolia costs, I decided to archive jobs older than 30 days by saving them to Amazon S3. These collections are just JSON files, so they could be re-imported into Algolia via their web interface or API at any time. The archival command is configurable to run any number of days back that you’d like.

Setting up the project

I’ve been running the project in Docker, and have documented it in the repository on Github. I welcome any additional input for documentation or support in the Github issues, but my time to update them will likely be limited going forward.

Several years ago when I started the JobAPIs project, job boards were competing to get their jobs in front of as many people and developers as possible. Since then, there has been a lot of consolidation in the market (SimplyHired, another big player shut down their whole site in 2016), and some job boards have started holding their data more closely. With Google entering the job aggregation space, I think there’s a lot of fear over where existing businesses will go in the next few years, but it seems that Dice has decided their business will go in a direction less friendly to developers.

As of October, 2017 the Dice Jobs API has been shut down. Dice has given me no reason for this, but there’s nothing I can do about it. The open source Dice Jobs library we support no longer works, but I’m keeping it on Github and Packagist to prevent breaking anyone’s dependencies. This is the response I got from Dice when I asked about it:

1
2
3
4
5

Thank you for contacting us. Unfortunately Dice no longer has a public API. 
If you have any questions, or if I may be of any further assistance, please respond to this email or call me directly for faster resolution.
Thank you for choosing Dice and have a great day!

One user reported to me that he had a conversation with Dice about this, and he said that their API was never intended to be used by external developers anyway. Nevertheless, it was documented and at least 870 projects used the Dice job board adapter we released.

That’s pretty much all I know about the shutdown at this point, but you’re welcome to reach out if you have more information. To contact Dice, you can fill out this form on their website.

A few months ago, we launched a job board aggregation service called Jobs Hub. After trialing the product with a few customers in the Spring and Summer of 2017, we were able to gather feedback, and improve the product, which has now been re-released as a self-service API.

What does that mean?

Instead of setting up a custom job board aggregator for each customer, we’ve generalized the product so that anyone can gain access to our database of tens of thousands of jobs that we collect every month. Our API is a developer-first interface that allows you to use our job board data for your site’s backfill, research project, or sales lead purposes.

New pricing for Jobs Hub

Because Jobs Hub doesn’t require a custom implementation, we were able to dramatically decrease the pricing. You can request access for a free API key with up to 100 queries per month, or get a paid monthly plan starting at just $20/month. For that low price, you’re able to gain access to tens of thousands of listings from most of the internet’s major job boards and aggregators!

To learn more, check out Jobs Hub or email me at admin@jobapis.com to request a free trial API key.

For the past three years, JobApis has been primarily focused on building open source software surrounding job boards and accessing job board data. We’ve gotten to work on a lot of interesting projects, and made some exciting contributions to the open source community.

After fielding hundreds of questions via email and Github, we’ve started to notice some trends. The most striking was that a number of non-developers were finding our code and then contacting us to ask for custom implementations or recommendations for developers who could help them build something with our open source tools.

At this point, supporting every user with one-on-one help isn’t possible, but this week we’re announcing the release of our first software as a service product for developers and non-developers who want to use job board data in their next venture:

What is Jobs Hub?

Jobs Hub is a tool that collects job listings and associated data from job boards and aggregators around the internet. You can read more on the product’s page, but with it, you can do things like:

• Monitor the web for new jobs matching specific search criteria and locations
• Build niche job board backfill
• Create custom job search services for clients or customers

Let’s say you want to keep up with all the companies hiring Ruby Developers in Chicago, IL. Jobs Hub will pull in all the listings it can find every day and allow you to search them in real-time as they’re available via our simple JSON API.

Jobs Hub builds on the open source technology that we support, so while we still encourage developers to build tools of their own, Jobs Hub will save them a lot of time.

More use cases and demos of Jobs Hub will be coming soon, but if you’d like to learn more or get a quote, contact Karl at admin@jobapis.com with some information about your project.

Jobs2Careers support has been on hold since last year’s release of Jobs Common v2. Thanks to some help from Samer Bechara we have gotten the package up and running again. It is now supported by Jobs Multi as well.

Get started using this API client by including the package in your project’s composer file:

1

composer require jobapis/jobs-jobs2careers

As with all Jobs Common providers, you first need to create a Query:

1
2
3
4
5
6

$query = new JobApis\Jobs\Client\Queries\J2cQuery([
    'id' => YOUR_PUBLISHER_ID,
    'pass' => YOUR_PUBLISHER_PASSWORD,
    'q' => 'engineer',
    'l' => 'Chicago, IL',
]);

And inject it into the J2cProvider:

1
2

$client = new JobApis\Jobs\Client\Providers\J2cProvider($query);
$jobs = $client->getJobs();

This will return a Collection of Job objects.

For more detailed usage and documentation, see this project’s readme, and as always, feel free to contact us if you’d like to get involved.

This month, we released two new job board clients: CareersInGovernment.com (Github) and GoRemote.io (Github). Both of these clients use RSS feeds to get the latest job listings.

Usage for each is pretty simple as there are no parameters that can be injected in the Query Objects.

Careers in Government

Install the package with Composer, and require the vendor file in your project. After the package is installed, you can get the latest listings in a couple lines of code:

1
2

$client = new CareersInGovProvider(new CareersInGovQuery());
$jobs = $client->getJobs();

For more detailed documentation see this project’s readme file.

GoRemote

Install the package with Composer, and require the vendor file in your project. After the package is installed, you can get the latest listings in a couple lines of code:

1
2

$client = new GoRemoteProvider(new GoRemoteQuery());
$jobs = $client->getJobs();

For more detailed documentation see this project’s readme file.

We now support PHPJobs.com’s RSS feed of job using the Jobs Common framework. With this client, you can quickly access the job search RSS feed on PHPJobs.com in your projects. Here’s an example of the library in action:

Usage

Install the package using Composer:

1

composer require jobapis/jobs-phpjobs

Then create a query object with all the parameters you’d like to use:

1
2
3
4

// Add parameters to the query via the constructor
$query = new JobApis\Jobs\Client\Queries\PhpjobsQuery([
    'search_string' => 'zend'
]);

And inject the query object into the provider:

1

$client = new JobApis\Jobs\Client\Provider\CraigslistProvider($query);

Finally, call the “getJobs” method to retrieve results:

1
2

// Get a Collection of Jobs
$jobs = $client->getJobs();

Check out more examples or contribute to the project on Github.

Today we released our 17th job board client using version 2.0 of the Jobs Common framework, a Craigslist job search RSS feed client. Using this client, you can quickly access the job search RSS feed on Craigslist in your PHP projects. Here’s an example of the library in action:

Usage

Install the package using Composer:

1

composer require jobapis/jobs-craigslist

Then create a query object with all the parameters you’d like to use:

1
2
3
4
5
6

// Add parameters to the query via the constructor
$query = new JobApis\Jobs\Client\Queries\CraigslistQuery([
    'location' => 'chicago',    // Craigslist location. Full list of locations here: https://sites.google.com/site/clsiteinfo/city-site-code-sort
    'query' => 'sales',         // Search query
    's' => '100',               // Offset results
]);

And inject the query object into the provider:

1

$client = new JobApis\Jobs\Client\Provider\CraigslistProvider($query);

Finally, call the “getJobs” method to retrieve results:

1
2

// Get a Collection of Jobs
$jobs = $client->getJobs();

Check out more examples or contribute to the project on Github.

Monster.com Jobs RSS feed client

Monster.com’s official API is closed to most users (they have not responded to multiple requests for access), but their RSS feed provides a small subset of their jobs. Unfortunately, results are only from the past 24 hours, so this client won’t work for all applications. Using the client is pretty much the same as most RSS feed providers we support:

1. Install the package via Composer

composer require jobapis/jobs-monster

2. Create a query object with all API parameters

1
2
3
4
5

$query = new JobApis\Jobs\Client\Queries\MonsterQuery([
    'q' => 'engineering', // Keyword
    'where' => 'chicago', // Location
    'p' => '1', // Page number
]);

3. Inject the query object and get jobs

1
2

$client = new JobApis\Jobs\Client\Provider\MonsterProvider($query);
$jobs = $client->getJobs();

This will return a Collection of Job objects.

For more detailed usage and documentation, see this project’s readme

The Muse Jobs API client

We also re-released our Muse API client with support for Jobs Common v.2 and the Muse’s v.2 of their API. Their API still does not support a true “keyword” search, but the category field allows you to search by types of jobs. Usage is similar to the above client:

1. Install the package via Composer

composer require jobapis/jobs-muse

2. Create a query object with all API parameters

1
2
3
4

$query = new JobApis\Jobs\Client\Queries\MuseQuery([
    'category' => 'Engineering',
    'location' => 'Chicago, IL',
]);

3. Inject the query object and get jobs

1
2

$client = new JobApis\Jobs\Client\Provider\MuseProvider($query);
$jobs = $client->getJobs();

This will return a Collection of Job objects.

For more detailed usage and documentation, see this project’s readme

Just in time for 2017, I got my act together and cleaned up things in JobsToMail for a version 1.0.0 release. There were a lot of major changes in the project since the last release, but I’ve had most of them in production for weeks now, so I’m not too concerned with their stability.

Release Highlights

• Support for 13 total job boards.
• User login via token, allowing users to view and manage job searches.
• Users can download their job listings every day in a convenient CSV file.
• Filtering recruiter job listings from our database of recruiting firms.
• Support for PHP 7.1.

See all release notes on Github.

What’s next for JobsToMail?

I’m actually going to be doing less work on JobsToMail for the next few months. It’s been a really fun project, and I’m glad it’s been useful to a few job seekers and interested developers, but JobApis as an organization has other priorities for 2017.

That said, I will continue to support the project and keep the servers running, and I encourage any developers who want to continue building it to keep the PRs coming.

Special thanks to Soleo, Alex Knoll, and Max Lever for their contributions to version 1.0. It’s been really fun working on this project with you guys and I hope to continue work in 2017!