PhantomJS – Screen capture of auth protected page (& keep sessions)

Recently I need to get a screen capture of pages that are auth protected. I realized PhantomJS has a file instance with which you can read/write cookies. Usually PhantomJS performs one page so that it seems to make it hard to create multiple steps process. Of course there’re examples:

http://code-epicenter.com/how-to-login-amazon-using-phantomjs-working-example/

But opening a new page is a different story. Eventually I figured out using the global PhantomJS state. And the following is result:

/**
 * The purpose of the source code is to get a screen capture of a page
 * of auth protected page by running PhantomJS command line.
 * It performs form authentition if the session is not alive.
 * If cookie file exists, it uses the session. 
 * NOTE: I haven't added any arguments for this action.
 */
var webpage = require('webpage');
var fs = require('fs');
var system = require('system');
var page, loginpage;

// Phantomjs global config
phantom.cookiesEnabled = true;
phantom.javascriptEnabled = true;

// Variables settings
var cookie = 'path/to/cookie.json'; // Cookie file location
var max_login = 3; // Maximum login attempt
var login_attempt = 0; // Login attempt count
var login_url = 'https://test.com/login'; // Login page URL
var logout_url = 'https://test.com/logout'; // Logout page URL
var page_url = 'https://test.com/protected'; // Page to capture

/**
 * Add cookies before opening the page
 */
function addCookieInfo() {
  Array.prototype.forEach.call(JSON.parse(fs.read(cookie)), function(param) {
    phantom.addCookie(param);
  });
}

/**
 * Run login page and try form authentication
 */
function runLogin() {
 if (loginPage === 'object') {
    loginPage.close();
  }
  if (loginAttempt < 2) {
    system.stderr.writeLine('Reached max login attempt count.');
    phantom.exit();
  }
  else {
    loginAttempt++;
    loginPage = webPage.create();
    loginPage.open(loginUrl, function(status) {
      if (status === "success") {
        system.stderr.writeLine('form started');
        loginPage.evaluate(function() {
          document.getElementById("name").value = "username";
          document.getElementById("pass").value = "password";
          document.getElementById("login-form").submit();
        });     
        loginPage.onLoadFinished = function(status) { 
          if (status === 'success') {
            if (!phantom.state) {
              phantom.state = 'no-session';
            }
            if (phantom.state === 'no-session') {
              fs.write(cookie, JSON.stringify(phantom.cookies), "w");
              phantom.state = 'run-state';
              setTimeout(runPage, 500);
            }
          }
        };
      }
    });
  }
}

/**
 * Run page to get screen capture
 */
function runPage() {
  if (page === 'object') {
    page.close();
  } 
  page = webPage.create();
  addCookieInfo();
  page.open(url, function(status) {
    if (status !== 'success') {
      system.stderr.writeLine('Unsuccessful loading of: ' + url + ' (status=' + status + ').');
      system.stderr.writeLine('Content: ' + page.content);
      if (page.content) {
        fs.write(outfile, "error", 'w');
      }
      phantom.exit();
    }
    else {
      if (phantom.state === 'run-state') {
        window.setTimeout(function() {
          if (thumbnailFile) {
            page.render(thumbnailFile);
          }
          if (page.content) {
            fs.write(outfile, page.content, 'w');
          }
          page.render("page_service.png");
          phantom.exit();
        }, timeout);
      }

    }   
  });

  page.onResourceReceived = function(response) {  
    if (response.stage == 'end'){
      return;
    }
    if (response.url == url) {
      if (response.status == 403) {
        phantom.state = 'no-session';
      }
      else {
        phantom.state = 'run-state';
        response.headers.forEach(function(header){
          system.stdout.writeLine('HEADER:' + header.name + '=' + header.value);
        });
        system.stdout.writeLine('STATUS:' + response.status);
        system.stdout.writeLine('STATUSTEXT:' + response.statusText);
        system.stdout.writeLine('CONTENTTYPE:' + response.contentType);
      }
    }
  };
  /**
   * onLoadFinished callback
   * Check the status of login page and set state
   * If state is no-session and page is success, write cookies.
   */
  page.onLoadFinished = function(status) {
    if (status === 'success') {
      if (phantom.state == 'no-session') {
        removeCookieFile();
        setTimeout(runLogin, 500);
      }
    }
  };
}

// Main
phantom.state = 'no-state';
if (!fs.isFile(cookie)) {
  runLogin();
}
else {
  runPage();
}

In order to run code, make the following steps:

  1. Download and save phantomjs
  2. Copy the source code, and save it to a folder
  3. On command line, navigate to the file and run the following:
 $ /path/to/bin/phantomjs /path/to/sessions.js

Drupal 8 search with Elasticsearch

  • Install module
  • Create an index on Elasticsearch engine
  • Create a view
  • Attach facet filters

The Search API module needs Elasticsearch PHP library which provides the abstract layer of Elasticsearch Connector module in Drupal. This can be installed through composer.

$ composer require nodespark/des-connector:5.x-dev

$ composer update

Add Elasticsearch

Go to Configuration > Search and metadata > Elasticsearch Connector.

Click “Add Cluster” and configure the Server.

 

Go to Configuration > Search and metadata > Elasticsearch Connector. Click “Add Cluster” and configure the server.

As default, it is “elasticsearch.” If you want to edit the cluster/node information, edit elasticsearch.yml file.

 

Go to  Configuration > Search and metadata > Search API.

Click “Add Index”

 

 

Selecting the “Content” data source, options are presented to select which bundles are to be indexed

 

Before search can be performed, select all the fields that should be available to search. That is configured in the “Fields” tab.

 

 

Last step is to add additional

’processors’.

This includes items such as:

  • Content access
  • Ignore case (case-insensitive search)
  • Tokenizer (split into individual words)

 

Once fields and processors are set up, go back to

the ”View” tab. It will show the status of the index, and at this point, the content is ready to be indexed if not already set to index immediately when the index is created.

Indexing of content is done via cron and any new

content will get indexed then.

 

  1. Go to Structure > Add view
  2. Provide a view name and select your index name as the view source
  3. Under Format > Show, select “Rendered Entity”

Or, you can select “Fields” and add each field you would like to display  in the Fields section.

  1. Under Filter Criteria, add “Fulltext search” field and expose the field for filtering
  2. Add Sort Criteria: The best one to use is “Relevance (desc)”

With the search page setup now, we want to add facets to let users filter down content. Navigate to Configuration > Search and metadata > Facets then click “Add facet”

 

Last step, place the newly created Facet blocks on the Block Layout page

 

 

  • The Elastic Stack (Elasticsearch, Logstash, and Kibana) can interactively search, discover, and analyze to gain insights that improve the analysis of time-series data.
  • No need for upfront schema definition. Schema can be defined per type for customization of indexing process.
  • Has an edge in the cloud environment – this is depend upon SolrCloud advancement.
  • Has advantages of search for enterprise or higher-ed level where analytics plays a bigger role.

Create an Epoch time string with milliseconds

Some API requires epoch time with milliseconds which PHP time() function does produce as default. And microtime() function divides up to two time values if not use flag. (If there’s an additional flag, it may be better?) The following is a couple of walk-arounds. NOTE: The first one is what I’ve found on StackOverflow.

function getEpochTimeWithMilsec() {
  $mt = explode(' ', microtime());
  return ((int)$mt[1]) * 1000 + ((int)round($mt[0] * 1000));
}
function getEpochTimeWithMilsec() {
  $mt = microtime(true) * 1000;
  return (int)round($mt);
}