[Curl] Implement functions to use ResourceLoadStatistics.

https://bugs.webkit.org/show_bug.cgi?id=207692

Patch by Takashi Komori <[email protected]> on 2020-06-12
Reviewed by Don Olmstead.

Implement functions which are required to implement ResourceLoadStatistics for Curl port.

Source/WebCore:

Tests: http/tests/resourceLoadStatistics/

* CMakeLists.txt:
* platform/network/curl/CookieJarDB.cpp:
(WebCore::CookieJarDB::openDatabase):
(WebCore::CookieJarDB::setCookie):
(WebCore::CookieJarDB::allDomains):
(WebCore::CookieJarDB::deleteCookiesForHostname):
* platform/network/curl/CookieJarDB.h:
* platform/network/curl/NetworkStorageSessionCurl.cpp:
(WebCore::NetworkStorageSession::setCookiesFromDOM const):
(WebCore::NetworkStorageSession::setCookies):
(WebCore::NetworkStorageSession::deleteCookiesForHostnames):
(WebCore::NetworkStorageSession::getHostnamesWithCookies):

Source/WebKit:

In NetworkDataTaskCurl.cpp we check if we should block cookies and block if needed.

Tests: http/tests/resourceLoadStatistics/

* NetworkProcess/NetworkDataTask.cpp:
(WebKit::NetworkDataTask::create):
* NetworkProcess/curl/NetworkDataTaskCurl.cpp:
(WebKit::NetworkDataTaskCurl::NetworkDataTaskCurl):
(WebKit::NetworkDataTaskCurl::createCurlRequest):
(WebKit::NetworkDataTaskCurl::willPerformHTTPRedirection):
(WebKit::NetworkDataTaskCurl::blockCookies):
(WebKit::NetworkDataTaskCurl::unblockCookies):
(WebKit::NetworkDataTaskCurl::shouldBlockCookies):
(WebKit::NetworkDataTaskCurl::isThirdPartyRequest):
* NetworkProcess/curl/NetworkDataTaskCurl.h:
* NetworkProcess/curl/NetworkSessionCurl.cpp:
(WebKit::NetworkSessionCurl::NetworkSessionCurl):
* UIProcess/API/C/WKWebsiteDataStoreRef.cpp:
(WKWebsiteDataStoreGetAllStorageAccessEntries):
* UIProcess/API/C/WKWebsiteDataStoreRef.h:
* UIProcess/WebsiteData/WebsiteDataStore.cpp:
(WebKit::WebsiteDataStore::parameters):

Tools:

Tests: http/tests/resourceLoadStatistics/

* WebKitTestRunner/TestController.cpp:
(WTR::GetAllStorageAccessEntriesCallbackContext::GetAllStorageAccessEntriesCallbackContext):
(WTR::getAllStorageAccessEntriesCallback):
(WTR::TestController::getAllStorageAccessEntries):

LayoutTests:

Tests: http/tests/resourceLoadStatistics/

* platform/wincairo-wk1/TestExpectations:
* platform/wincairo/TestExpectations:

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@262971 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Author: [email protected]

LayoutTests/ChangeLog

@@ -1,3 +1,17 @@
+2020-06-12  Takashi Komori  <[email protected]>
+
+        [Curl] Implement functions to use ResourceLoadStatistics.
+        https://bugs.webkit.org/show_bug.cgi?id=207692
+
+        Reviewed by Don Olmstead.
+
+        Implement functions which are required to implement ResourceLoadStatistics for Curl port.
+
+        Tests: http/tests/resourceLoadStatistics/
+
+        * platform/wincairo-wk1/TestExpectations:
+        * platform/wincairo/TestExpectations:
+
 2020-06-12  Yusuke Suzuki  <[email protected]>
 
         [ Mojave wk2 Release ] js/dom/unhandled-promise-rejection-console-no-report.html is flaky failing.

LayoutTests/platform/wincairo-wk1/TestExpectations

@@ -309,3 +309,6 @@ webkit.org/b/191498 http/tests/inspector/network/getSerializedCertificate.html [
 
 # Skip IsLoggedIn
 http/tests/is-logged-in/ [ Skip ]
+
+# Skip ResourceLoadStatistics
+http/tests/resourceLoadStatistics [ Skip ]

LayoutTests/platform/wincairo/TestExpectations

@@ -1175,9 +1175,37 @@ http/tests/referrer-policy/no-referrer-when-downgrade/ [ Skip ]
 http/tests/referrer-policy/unsafe-url/cross-origin-http-http.html [ Skip ] 
 http/tests/referrer-policy/unsafe-url/cross-origin-http.https.html [ Skip ] 
 
-# All timing out
-http/tests/resourceLoadStatistics [ Skip ]
+# ResourceLoadStatistics
+http/tests/resourceLoadStatistics [ Pass ]
+http/tests/resourceLoadStatistics/exemptDomains [ Skip ]
+http/tests/resourceLoadStatistics/add-blocking-to-redirect-database.html [ Skip ]
+http/tests/resourceLoadStatistics/add-blocking-to-redirect.html [ Skip ]
+http/tests/resourceLoadStatistics/cookie-deletion-database.html [ Skip ]
+http/tests/resourceLoadStatistics/cookie-deletion.html [ Skip ]
+http/tests/resourceLoadStatistics/cookies-with-and-without-user-interaction-database.html [ Skip ]
+http/tests/resourceLoadStatistics/cookies-with-and-without-user-interaction.html [ Skip ]
+http/tests/resourceLoadStatistics/do-not-remove-blocking-in-redirect-database.html [ Skip ]
+http/tests/resourceLoadStatistics/do-not-remove-blocking-in-redirect.html [ Skip ]
+http/tests/resourceLoadStatistics/enforce-samesite-strict-based-on-top-frame-unique-redirects-to-database.html [ Skip ]
+http/tests/resourceLoadStatistics/enforce-samesite-strict-based-on-top-frame-unique-redirects-to.html [ Skip ]
+http/tests/resourceLoadStatistics/ping-to-prevalent-resource-database.html [ Skip ]
+http/tests/resourceLoadStatistics/ping-to-prevalent-resource.html [ Skip ]
+http/tests/resourceLoadStatistics/set-all-cookies-to-same-site-strict.html [ Skip ]
+http/tests/resourceLoadStatistics/standalone-web-application-exempt-from-website-data-deletion-database.html [ Skip ]
+http/tests/resourceLoadStatistics/standalone-web-application-exempt-from-website-data-deletion.html [ Skip ]
+http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-database.php [ Skip ]
+http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction.php [ Skip ]
+http/tests/resourceLoadStatistics/telemetry-generation.html [ Skip ]
+http/tests/resourceLoadStatistics/third-party-cookie-blocking-ephemeral-database.html [ Skip ]
+http/tests/resourceLoadStatistics/third-party-cookie-blocking-ephemeral.html [ Skip ]
+http/tests/resourceLoadStatistics/third-party-cookie-blocking-on-sites-without-user-interaction-database.html [ Skip ]
+http/tests/resourceLoadStatistics/third-party-cookie-blocking-on-sites-without-user-interaction.html [ Skip ]
+http/tests/resourceLoadStatistics/user-interaction-only-reported-once-within-short-period-of-time-database.html [ Skip ]
+http/tests/resourceLoadStatistics/user-interaction-only-reported-once-within-short-period-of-time.html [ Skip ]
+http/tests/resourceLoadStatistics/user-interaction-reported-after-website-data-removal-database.html [ Skip ]
+http/tests/resourceLoadStatistics/user-interaction-reported-after-website-data-removal.html [ Skip ]
 
+# All timing out
 http/tests/security [ Skip ]
 http/tests/security/401-logout/401-logout.php [ Pass ]
 http/tests/security/cookies/third-party-cookie-blocking.html [ Pass ]

Source/WebCore/CMakeLists.txt

@@ -696,6 +696,7 @@ set(WebCore_NON_SVG_IDL_FILES
     dom/DocumentFragment.idl
     dom/DocumentFullscreen.idl
     dom/DocumentOrShadowRoot.idl
+    dom/DocumentStorageAccess.idl
     dom/DocumentTouch.idl
     dom/DocumentType.idl
     dom/DragEvent.idl

Source/WebCore/ChangeLog

@@ -1,3 +1,27 @@
+2020-06-12  Takashi Komori  <[email protected]>
+
+        [Curl] Implement functions to use ResourceLoadStatistics.
+        https://bugs.webkit.org/show_bug.cgi?id=207692
+
+        Reviewed by Don Olmstead.
+
+        Implement functions which are required to implement ResourceLoadStatistics for Curl port.
+
+        Tests: http/tests/resourceLoadStatistics/
+
+        * CMakeLists.txt:
+        * platform/network/curl/CookieJarDB.cpp:
+        (WebCore::CookieJarDB::openDatabase):
+        (WebCore::CookieJarDB::setCookie):
+        (WebCore::CookieJarDB::allDomains):
+        (WebCore::CookieJarDB::deleteCookiesForHostname):
+        * platform/network/curl/CookieJarDB.h:
+        * platform/network/curl/NetworkStorageSessionCurl.cpp:
+        (WebCore::NetworkStorageSession::setCookiesFromDOM const):
+        (WebCore::NetworkStorageSession::setCookies):
+        (WebCore::NetworkStorageSession::deleteCookiesForHostnames):
+        (WebCore::NetworkStorageSession::getHostnamesWithCookies):
+
 2020-06-12  Andres Gonzalez  <[email protected]>
 
         In isolated tree mode 2, AXIsolatedObject::setChildrenIDs should be called only on secondary thread.

Source/WebCore/platform/network/curl/CookieJarDB.cpp

@@ -24,6 +24,7 @@
 
 #include "config.h"
 #include "CookieJarDB.h"
+#include "cookieJar.h"
 
 #include "CookieUtil.h"
 #include "Logging.h"
@@ -63,6 +64,8 @@ namespace WebCore {
     "CREATE INDEX IF NOT EXISTS domain_index ON Cookie(domain);"
 #define CREATE_PATH_INDEX_SQL \
     "CREATE INDEX IF NOT EXISTS path_index ON Cookie(path);"
+#define SELECT_ALL_DOMAINS_SQL \
+    "SELECT DISTINCT domain FROM Cookie;"
 #define CHECK_EXISTS_COOKIE_SQL \
     "SELECT domain FROM Cookie WHERE ((domain = ?) OR (domain GLOB ?));"
 #define CHECK_EXISTS_HTTPONLY_COOKIE_SQL \
@@ -76,6 +79,10 @@ namespace WebCore {
     "DELETE FROM Cookie WHERE name = ? AND domain = ?;"
 #define DELETE_ALL_SESSION_COOKIE_SQL \
     "DELETE FROM Cookie WHERE session = 1;"
+#define DELETE_COOKIES_BY_DOMAIN_SQL \
+    "DELETE FROM Cookie WHERE domain = ? ;"
+#define DELETE_COOKIES_BY_DOMAIN_EXCEPT_HTTP_ONLY_SQL \
+    "DELETE FROM Cookie WHERE (domain = ?) AND (httponly = 0);"
 #define DELETE_ALL_COOKIE_SQL \
     "DELETE FROM Cookie;"
 
@@ -165,6 +172,8 @@ bool CookieJarDB::openDatabase()
     createPrepareStatement(CHECK_EXISTS_HTTPONLY_COOKIE_SQL);
     createPrepareStatement(DELETE_COOKIE_BY_NAME_DOMAIN_PATH_SQL);
     createPrepareStatement(DELETE_COOKIE_BY_NAME_DOMAIN_SQL);
+    createPrepareStatement(DELETE_COOKIES_BY_DOMAIN_SQL);
+    createPrepareStatement(DELETE_COOKIES_BY_DOMAIN_EXCEPT_HTTP_ONLY_SQL);
 
     return true;
 }
@@ -529,7 +538,7 @@ bool CookieJarDB::setCookie(const Cookie& cookie)
     return checkSQLiteReturnCode(statement.step());
 }
 
-bool CookieJarDB::setCookie(const URL& firstParty, const URL& url, const String& body, CookieJarDB::Source source)
+bool CookieJarDB::setCookie(const URL& firstParty, const URL& url, const String& body, CookieJarDB::Source source, Optional<Seconds> cappedLifetime)
 {
     if (!isEnabled() || !m_database.isOpen())
         return false;
@@ -550,9 +559,31 @@ bool CookieJarDB::setCookie(const URL& firstParty, const URL& url, const String&
     if (!canAcceptCookie(*cookie, firstParty, url, source))
         return false;
 
+    if (cappedLifetime && cookie->expires) {
+        ASSERT(*cappedLifetime >= 0_s);
+        auto cappedExpires = WallTime::now() + *cappedLifetime;
+        if (cappedExpires < WallTime::fromRawSeconds(*cookie->expires / WTF::msPerSecond))
+            cookie->expires = cappedExpires.secondsSinceEpoch().milliseconds();
+    }
+
     return setCookie(*cookie);
 }
 
+HashSet<String> CookieJarDB::allDomains()
+{
+    SQLiteStatement statement(m_database, SELECT_ALL_DOMAINS_SQL);
+    statement.prepare();
+
+    HashSet<String> domains;
+    while (statement.step() == SQLITE_ROW) {
+        auto domain = statement.getColumnText(0);
+        domains.add(domain);
+    }
+
+    statement.finalize();
+    return domains;
+}
+
 bool CookieJarDB::deleteCookie(const String& url, const String& name)
 {
     if (!isEnabled() || !m_database.isOpen())
@@ -589,6 +620,13 @@ bool CookieJarDB::deleteCookies(const String&)
     return false;
 }
 
+bool CookieJarDB::deleteCookiesForHostname(const String& hostname, IncludeHttpOnlyCookies includeHttpOnlyCookies)
+{
+    auto& statement = preparedStatement(includeHttpOnlyCookies == IncludeHttpOnlyCookies::Yes ? DELETE_COOKIES_BY_DOMAIN_SQL : DELETE_COOKIES_BY_DOMAIN_EXCEPT_HTTP_ONLY_SQL);
+    statement.bindText(1, hostname);
+    return checkSQLiteReturnCode(statement.step());
+}
+
 bool CookieJarDB::deleteAllCookies()
 {
     if (!isEnabled() || !m_database.isOpen())

Source/WebCore/platform/network/curl/CookieJarDB.h

@@ -25,6 +25,7 @@
 #pragma once
 
 #include "Cookie.h"
+#include "CookieJar.h"
 #include "SQLiteDatabase.h"
 #include "SQLiteStatement.h"
 #include <wtf/HashMap.h>
@@ -56,13 +57,15 @@ class CookieJarDB {
     void setAcceptPolicy(CookieAcceptPolicy policy) { m_acceptPolicy = policy; }
     CookieAcceptPolicy acceptPolicy() const { return m_acceptPolicy; }
 
+    HashSet<String> allDomains();
     Optional<Vector<Cookie>> searchCookies(const URL& firstParty, const URL& requestUrl, const Optional<bool>& httpOnly, const Optional<bool>& secure, const Optional<bool>& session);
     Vector<Cookie> getAllCookies();
-    bool setCookie(const URL& firstParty, const URL&, const String& cookie, Source);
+    bool setCookie(const URL& firstParty, const URL&, const String& cookie, Source, Optional<Seconds> cappedLifetime = WTF::nullopt);
     bool setCookie(const Cookie&);
 
     bool deleteCookie(const String& url, const String& name);
     bool deleteCookies(const String& url);
+    bool deleteCookiesForHostname(const String& hostname, IncludeHttpOnlyCookies);
     bool deleteAllCookies();
 
     WEBCORE_EXPORT CookieJarDB(const String& databasePath);

Source/WebCore/platform/network/curl/NetworkStorageSessionCurl.cpp

@@ -95,9 +95,14 @@ CookieJarDB& NetworkStorageSession::cookieDatabase() const
     return m_cookieDatabase;
 }
 
-void NetworkStorageSession::setCookiesFromDOM(const URL& firstParty, const SameSiteInfo&, const URL& url, Optional<FrameIdentifier>, Optional<PageIdentifier>, ShouldAskITP, const String& value, ShouldRelaxThirdPartyCookieBlocking) const
+void NetworkStorageSession::setCookiesFromDOM(const URL& firstParty, const SameSiteInfo&, const URL& url, Optional<FrameIdentifier> frameID, Optional<PageIdentifier> pageID, ShouldAskITP, const String& value, ShouldRelaxThirdPartyCookieBlocking shouldRelaxThirdPartyCookieBlocking) const
 {
-    cookieDatabase().setCookie(firstParty, url, value, CookieJarDB::Source::Script);
+#if ENABLE(RESOURCE_LOAD_STATISTICS)
+    Optional<Seconds> cappedLifetime = clientSideCookieCap(RegistrableDomain { firstParty }, pageID);
+#else
+    Optional<Seconds> cappedLifetime = WTF::nullopt;
+#endif
+    cookieDatabase().setCookie(firstParty, url, value, CookieJarDB::Source::Script, cappedLifetime);
 }
 
 void NetworkStorageSession::setCookiesFromHTTPResponse(const URL& firstParty, const URL& url, const String& value) const
@@ -133,9 +138,10 @@ std::pair<String, bool> NetworkStorageSession::cookiesForDOM(const URL& firstPar
     return { cookiesForSession(*this, firstParty, url, false), false };
 }
 
-void NetworkStorageSession::setCookies(const Vector<Cookie>&, const URL&, const URL& /* mainDocumentURL */)
+void NetworkStorageSession::setCookies(const Vector<Cookie>& cookies, const URL&, const URL& /* mainDocumentURL */)
 {
-    // FIXME: Implement for WebKit to use.
+    for (const auto& cookie : cookies)
+        cookieDatabase().setCookie(cookie);
 }
 
 void NetworkStorageSession::setCookie(const Cookie& cookie)
@@ -164,9 +170,10 @@ void NetworkStorageSession::deleteAllCookiesModifiedSince(WallTime)
     // FIXME: Not yet implemented
 }
 
-void NetworkStorageSession::deleteCookiesForHostnames(const Vector<String>& /* cookieHostNames */, IncludeHttpOnlyCookies)
+void NetworkStorageSession::deleteCookiesForHostnames(const Vector<String>& cookieHostNames, IncludeHttpOnlyCookies includeHttpOnlyCookies)
 {
-    // FIXME: Not yet implemented.
+    for (auto hostname : cookieHostNames)
+        cookieDatabase().deleteCookiesForHostname(hostname, includeHttpOnlyCookies);
 }
 
 void NetworkStorageSession::deleteCookiesForHostnames(const Vector<String>& cookieHostNames)
@@ -179,9 +186,9 @@ Vector<Cookie> NetworkStorageSession::getAllCookies()
     return cookieDatabase().getAllCookies();
 }
 
-void NetworkStorageSession::getHostnamesWithCookies(HashSet<String>& /* hostnames */)
+void NetworkStorageSession::getHostnamesWithCookies(HashSet<String>& hostnames)
 {
-    // FIXME: Not yet implemented
+    hostnames = cookieDatabase().allDomains();
 }
 
 Vector<Cookie> NetworkStorageSession::getCookies(const URL&)

Source/WebKit/ChangeLog

@@ -1,3 +1,35 @@
+2020-06-12  Takashi Komori  <[email protected]>
+
+        [Curl] Implement functions to use ResourceLoadStatistics.
+        https://bugs.webkit.org/show_bug.cgi?id=207692
+
+        Reviewed by Don Olmstead.
+
+        Implement functions which are required to implement ResourceLoadStatistics for Curl port.
+
+        In NetworkDataTaskCurl.cpp we check if we should block cookies and block if needed.
+
+        Tests: http/tests/resourceLoadStatistics/
+
+        * NetworkProcess/NetworkDataTask.cpp:
+        (WebKit::NetworkDataTask::create):
+        * NetworkProcess/curl/NetworkDataTaskCurl.cpp:
+        (WebKit::NetworkDataTaskCurl::NetworkDataTaskCurl):
+        (WebKit::NetworkDataTaskCurl::createCurlRequest):
+        (WebKit::NetworkDataTaskCurl::willPerformHTTPRedirection):
+        (WebKit::NetworkDataTaskCurl::blockCookies):
+        (WebKit::NetworkDataTaskCurl::unblockCookies):
+        (WebKit::NetworkDataTaskCurl::shouldBlockCookies):
+        (WebKit::NetworkDataTaskCurl::isThirdPartyRequest):
+        * NetworkProcess/curl/NetworkDataTaskCurl.h:
+        * NetworkProcess/curl/NetworkSessionCurl.cpp:
+        (WebKit::NetworkSessionCurl::NetworkSessionCurl):
+        * UIProcess/API/C/WKWebsiteDataStoreRef.cpp:
+        (WKWebsiteDataStoreGetAllStorageAccessEntries):
+        * UIProcess/API/C/WKWebsiteDataStoreRef.h:
+        * UIProcess/WebsiteData/WebsiteDataStore.cpp:
+        (WebKit::WebsiteDataStore::parameters):
+
 2020-06-12  Antoine Quint  <[email protected]>
 
         Double tap to zoom out doesn't work after upgrading to iOS 13

Source/WebKit/NetworkProcess/NetworkDataTask.cpp

@@ -60,7 +60,7 @@ Ref<NetworkDataTask> NetworkDataTask::create(NetworkSession& session, NetworkDat
     return NetworkDataTaskSoup::create(session, client, parameters.request, parameters.webFrameID, parameters.webPageID, parameters.storedCredentialsPolicy, parameters.contentSniffingPolicy, parameters.contentEncodingSniffingPolicy, parameters.shouldClearReferrerOnHTTPSToHTTPRedirect, parameters.isMainFrameNavigation);
 #endif
 #if USE(CURL)
-    return NetworkDataTaskCurl::create(session, client, parameters.request, parameters.storedCredentialsPolicy, parameters.contentSniffingPolicy, parameters.contentEncodingSniffingPolicy, parameters.shouldClearReferrerOnHTTPSToHTTPRedirect, parameters.isMainFrameNavigation);
+    return NetworkDataTaskCurl::create(session, client, parameters.request, parameters.webFrameID, parameters.webPageID, parameters.storedCredentialsPolicy, parameters.contentSniffingPolicy, parameters.contentEncodingSniffingPolicy, parameters.shouldClearReferrerOnHTTPSToHTTPRedirect, parameters.isMainFrameNavigation, parameters.shouldRelaxThirdPartyCookieBlocking);
 #endif
 }